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FOREWORD 



This research and development was conducted In response to Navy Decision 
Coordinating Paper, Performance Aids Test and Evaluation (NDCP-Z0828-PN) , 
under the sponsorship of the Director, Haval Education and Training (OP-99)* 

The overall objectives of the NDCP are to define the state-of-the-art 
in Job Performance Aids (JPA) technology, to develop a conceptual model 
for an integrated JPA-based personnel system including cost benefits and 
trade-off analysis, to test the JPA concept, and to quantify performance 
increments and cost benefits obtainable for various applications* The 
immediate effort described in this report relates to Task 1 of the NDCP: 

(1) the systematic review and organization of existing JPA techniques, re* 
lated research data, and various applicable principles and concepts, and 

(2) the establishment of meaningful discourse within the JPA reaearch, 
development, and operational conmmnities* The final two sectiona of the 
report, on conceptual organization and new research directiona, are directed 
to the JPA community at large* As such, comments are not only welcomed 

but solicited on the concepts and recommendations presented* 

Appreciation is expressed to Dr* John Foley of the Air Force Human Re- 
sources Laboratory, Dr* Edgar Shriver of Kinton, Inc., Mr* Theodore Post of 
BioTechnology, Inc* , and Mr* Tom Elliott of Applied Science Associates for 
their assistance in providing key documents that were useful in describing 
and organizing the existing JPA technology base* In addition, their con- 
cepts and those of others involved in performance aiding research, such aa 
Mr* Reid Joyce of Applied Science Associates and Mr* John Klesch of the 
Army, were extremely valuable in arriving at those preaented in thia report* 
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Commanding Officer 



SUMMARY 



Problem 

Data collected over the past 20 years, primarily by the military services 
but also in the civilian sector, suggest that Job Performance Aid (JPA) 
technology offers major potential payoffs in personnel performance incre- 
ments and cost avoidance throughout the personnel, training, and maintenance 
communities. Unfortunately, Che lack of a well- formula ted conceptual defini- 
tion of JPAs has impeded the advancement and utilization of JPA technology 
in military personnel, maintenance, and training programs. This has created 
difficulties in retaining researchers and practitioners in the field, has 
discouraged the development of a solid theoretical basis for JPAs, and has 
contributed to underutilization of the technology. Moreover, an integrated 
approach to JPAs in personnel systems has never been taken. 

Purpose 

The Navy Personnel Research and Development Center is conducting a major 
effort concerning the test and evaluation of JPAs and their integration with 
Navy personnel and training systems. That effort, outlined in Navy Decision 
Coordinating Paper, Performance Aids Test and Evaluation (NDCP-20828-PN) , 
has as its first task the detailed definition of current JPA technology. 
The objective of the present study, conducted as part of the larger task, 
was to systematically review and organize existing JPA techniques, related 
research data, and various applicable principles and concepts. 

Approach 

~A literature review and conceptual analysis were conducted to assess and 
organize the state-of-the-art of JPA research and technology. 

Results 

In the present study, 101 JPA systems and techniques were classified 
under five categories: (1) format/content, (2) display media, (3) applied 
training, (4) peripheral test/diagnostic, and (5) delivery systems. Major 
factors that were identified as critical to the development of a JPA algorithm 
included (1) personnel aptitude and experience, (2) type and complexity of 
task, (3) type and complexity of equipment, and (4) the degree of procedural!- 
zation required. A conceptual model was developed for use by the JPA com* 
munity in making cost trade-off analyses, writing JPA selection algorithms, 
and grouping theoretical trends. The study also examined a theoretical base 
for the use of memory in JPAs; previewed a theory for mixing JPA techniques, 
principles, and methodologies; and set goals for future JPA research and 
technology efforts. 

Conclusions 

1. Several format /content type JPA systems are sufficiently developed 
to be integrated with major military personnel and training systems with 
reasonable expectations of major payoffs in reduced personnel, training, 
and maintenance costs and in increased overall operational readiness. 
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2. Within format/content types, it appears that three — the FPJPA, the 
hybrid aid, and the deductive aid— would be sufficient to cover most opera- 
tion/maintainer aiding needs throughout their Navy careers. However, the 
hybrid aid requires further refinement before full-scale demonstration or 
implementation. 

3. Satisfactory aiding techniques exist for all non troubleshooting 
operation and maintenance tasks and for the majority of troubleshooting 
tasks on most mechanical, electrical, and electromechanical syatema, but 
no aiding techniques were found that were satisfactory for complex digital 
electronic systems, 

4. The greatest weakness in JPA systems technology is in the timely 
and accurate production, distribution, and update of JFAs. 

5. The major aid development cost items to be considered when com- 
paring one format /content type JPA with another are (a) the detail required 
for front-end analysis and (b) the amount of graphic or verbal enrichment 
necessary for increased intelligibility. 

6* The greater the amount of procedurallzation and enrichment, the 
greater the production, distribution, and storage costs becauae of increased 
aid size. 

7, Microform and audiovisual systems either show no differences or 
are inferior to hard-copy print media in terms of their effects on user per- 
formance. For information aiding needs, most of the unique demands of the 
job environment (e.g., work space, wind, temperature) can be met with modified 
hard-copy print. 

8, Devices that can increase the performance capability of the user 
beyond that possible with hard-copy print need further research and develop- 
ment. 

9, The FPJPA appears unsatisfactory for aiding lower-aptitude personnel 
to perform troubleshooting tasks, but it can allow inexperienced peraonnel 
who meet normal aptitude requirements to perform such tasks with minimal 
training. 

10, Existing JPA development and selection techniques hav; not provided 
sufficient organisation of the technology to permit confident reaearch or 
implementation decisions. 

11, Basic research is needed to answer questions about the behavioral 
foundation of aiding, 

12, There is still a need to determine the effects of aiding on lowered 
aptitude levels and the motivation to use aiding systems. 

13, Cost/performance criteria and basic cost data are needed to make 
valid comparisons of aids in trade-off models. 
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14. Research is needed to help determine what kind of process is re- 
quired to produce and measure high-quality JPAs. 



Recommendat Ions 

1. The methods and problems of integrated JPAs with training and job 
design should receive immediate attention in efforts directed toward algorithm 
development. 

2. Problems of career advancement, skills retention, and other major 
personnel system factors should be investigated to determine the effects of 
progressively changing JPA systems. 

3. Task-oriented training and job design should be developed to support 
or complement the aiding system. 

4. Aiding as a personnel system technology should act on an equal 
footing with selection and training, but it should be the primary technique 
for meeting most job objectives for new or first-term enlistees- 

5- Front-end analyses commonly conducted for integrated JPA and train- 
ing requirements should be expanded to include job design requirements* 

6. Selection of media other than hard-copy print should be based on 
system support requirements that improve the timeliness or accuracy of the 
aid rather than personnel performance-related criteria* 

7. Development of new aiding devices (e.g*, hand-held calculators, 
holographic displays), aids that change with personnel needs (e.g., hybrid 
aids), and aids that specifically address digital electronic systems are 
needed to fill gaps and to advance the state-of-the-art in aiding technology. 
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INTRODUCTION > 



Problem 

The Navy currently faces expanding personnel costs; decreasing force 
levels; declining entry-level skills; and degraded operational readiness 
with respect to personnel, systems, and equipment. Advances in personnel 
technology are therefore being c&mbed for systems or techniques to enhance 
the performance of lesser-trained or lesser-skilled individuals, to reduce 
the attrition of personnel with Navy career potential, and to reduce per- 
sonnel and training costs. 

Data collected over the past 20 years, primarily by the military, suggest 
that job performance aiding technology offers major potential in personnel 
performance increments and cost avoidance. Unfortunately, the lack of a 
well-formulated, conceptual definition of job performance aids (JPAs) has 
impeded the advancement and utilization of JPA technology in military per- 
sonnel, maintenance, and training programs. This has created difficulties 
in retaining researchers and practitioners in the field, has discouraged 
the development of a solid theoretical basis for JPAs, and has contributed 
to under utilization of the technology. 

Recently, users have perceived their major problem to be thfe* of select- 
ing from among the hundreds of JPA candidates, and researchers have obligingly 
directed most of their efforts toward the development of JPA selection algo- 
rithms. Little progress has been made in this area, however, because a 
total integrated approach to JPAs in personnel systems has never been taken 
(Booher, 1977a). Thus, the first steps for the development of a JPA-rele- 
vant integrated personnel system are (1) the formulation of a conceptual 
definition of JPAs and (2) an analysis of the state^of- thenar t in performance* 
aiding R&D. 

Purpose 

The Navy Personnel Research and Development Center is conducting a major 
effort concerning the test and evaluation of JPAs and their integration with 
Navy personnel and training systems. That effort, outlined in Navy Decision 
Coordinating Paper, Performance Aids Test and Evaluation (NDCP-Z0828-PN) , 
has as its first task the detailed definition of current JPA technology. 
Specific objectives of that task are (1) to conduct a comprehensive survey 
of current JPA systems and techniques, (2) to investigate implications for 
equipment, environmental, personnel, and behavioral characteristics of job 
tasks, and (3) to develop schema that would aid in achieving optimal utilisa- 
tion of state-of-the-art JPA technology. 

The purpose of the present study is to meet these objectives by systema- 
tically reviewing and organizing existing JPA systems and techniques; JPA 
research data; and related principles, methods, and concepts. 

Background 

In February 1977, NAVPERSRANDCEN sponsored a conference at which selected 
researchers and practitioners were asked to critically assess the current 
status of JPA technology (Booher, 1977a). A primary objective of that 



conference was to determine if JPA technology could be structured to stimu- 
late meaningful discourse among those concerned with JPA research, develop- 
ment, £nd implementation. It became apparent that, while such a structure, 
would be useful, ^here are a great many unique problems associated with 
JPA technology R&D* The conference's findings and the results of an analysis 
by NAVPERSRANDCEN provide the basis for the approach taken in this study. 

JPA Definition 

In the literature, the earliest definition of the JPA was expressed in 
terms of man-machine systems; 

A job aid is an item whose purpose is the support of per- 
formances by system personnel, which are necessary for over- 
all system performance. It is developed as an alternative 
to training as a means of obtaining the necessary perfor- 
mances, and is therefore designed deliberately to complement 
training. The job aid is developed in response t A a require- 
ment which specifies the performances that must be supported 
and the conditions under which these performances must be 
obtained. It is an end product of a system development cycle 
which is delivered as part of an operational system. (Wulff & t 
Berry, 1962,' p. 279) 

More recently, Smillie (Note 1) redefined the job aid more concisely as; 

Any contrivance which permits the human component of the 
system to perform some function which could not otherwise 
be accomplished without extensive training or complex infor- 
mation processing, (p. 1) 

Host often, definitions use information storage and retrieval language. 
For example^ Joyce (1975) said: 

The defining characteristic of a performance aid is the 
capacity to store information for later retrieval in con** 
nection with the performance of a, job. The aid facilitates 
performance by reducing the memory—and possibly training- 
requirements imposed upon the performer, (p. 5) 

Several definitions have emphasized proceduralization—typically mean- 
ing fully proceduralized job performance aids (FPJPAs) , which are "Documents 
or devices that give precise step-by-step instruction for each task or other- 
wise present in a concise and consolidated manner all information relevant 
to the task 11 (Rowan, 1973, p. 1). This is primarily a behavior-guiding or 
"active" definition in that it states the direct purpose of the aid in terms 
of supporting the efficient performance of a predetermined set of behaviors. 

Some researchers prefer to look also at the indirect or "passive" aspects 
of job aiding. Chalupsky and Kopf (1967), for example, addressed the infor- 
mational content and instructional aspects of job aids and how they facilitate 



performance by relieving certain job demands. They categorized these as 
task .modifying aids and cited as an example a table of square roots that 
performs the appropriate calculations for the user. 

An interesting theoretical approach to defining the jpa was taken by 
Lewis and Cook (1969) in considering the JPA an undirectlonal communica* 
tlon device. In describing a "theory of telling," they pointed out that, 
since there is no feedback from the user, the design of a JPA must be re- 
sponsive to a model of the user's abilities, limitations, and processing 
functions in attending, listening, and reading. They therefore inferred 
that'pictorial and linguistic channels are primary means of telling and 
that algorithms in the form of question-and-answer flow charts or question 
lists are superior to prose for this purpose. 

A rather unusual insight into the problem of defining JPAs is the recog- 
nition that it is not the media or format aspects of JPAs that affect per- 
formance aiding so much as it is the ability of the JPA to change the con* 
tent of the information. Shriver and Hart (1975) concluded that much of the 
research on new JPA concepts deals with the software link between the hard* 
ware and maintenance personnel. They recognised that, traditionally, this 
software link has been the technical manual. Although media (e.g., micro- 
fiche, tape, file records) have been substituted in the link, it has been 
informational content, and not the medium of presentation, that has changed 
job performance* They stressed that the new concepts of importance are those 
concerned with changing the information that flows across the link* 

Although aspects of all these definitions apply to JPAs, the following 
definition has been adopted for the purposes of the present research on 
performance enhancement: "A job performance aid is any device, manual, 
guide, or tool used on the job to facilitate performance or to avoid costs 
where learning from the aid is incidental." Such aids can vary from a simple 
metric conversion card to a complex troubleshooting logic network diagram 
used in electronic fault isolation. Other examples are a pilot's preflight 
checklist, a hand«-h'eld computer, or a job-guide manual. 

The written (textual) aid could be a fully proceduralized, step-by-step 
guide requiring no user decision making; it could provide only an overall 
picture of available options in various problem situations with deductive 
guidelines as to various decision paths; or it could be a combination of 
the two, directed toward an intermediate level of performance aiding* 

For tha most part, a JPA is a means of extending human capability by 
providing a mechanism . for storing information for later retrieval in con- 
nection with on-the-job performance. It also may serve as a short-term 
memory facilitator, since human memory often is not sufficiently reliable 
and because some tasks are performed too seldom for habit patterns to form* 
Consequently, an aid may reduce or eliminate training requiretoents by 
eliminating the need to learn and retain information concerning specific 
aspects of a job. 



DESCRIPTION AND COMPARISON OF JPA TECHNIQUES 



The glossary lists 101 JPA techniques that were surveyed in this study. 
Shriver and Hart (1975) reviewed many of them in the development of their 
"fundamental elements, 11 noting that many can be sorted into relatively inde- 
pendent classes. Post, Price, and Diffley (1976) also classified various 
JPAs into basic types of format and content for use in a guide fpr selecting 
formats and media in maintenance information presentation. Considerable 
thought has been given to the problem, but no general agreement has been 
reached among JPA investigators on a classification system that would be most 
useful in describing and comparing various JPA techniques. This study, there* 
fore, has sought to describe JPA technology in a global sense, considering 
aiding techniques not only individually but also in conjunction with other 
technologies and then breaking out the various classes of aids in a way that 
might help the decision-making activities of researchers, developers, and 
implementers. 

The conference on job aids (Booher, 1977a) concluded that the JPA tech- 
nology base can be organized into three fairly distinct categories: (I) 
technology associated directly with jok-aiding system*, which include** for- 
mat or content types, display media typesi, applied trailing systems, 
peripheral diagnostic systems, and delivery systems, {£) technology com- 
peting with job aids, which includes automatic test equipment, maintain 
ability design, formal training, selection* and human engineering design, 
and (3) technology supporting job aids, which includes instructional system 
design, job design, maintenance simulation, test and evaluation technology, 
and performance testing. This chapter simply describes and compares JPA 
systems, devices, and techniques that exemplify the five subcategories 
of direct job-aiding systems. The glossary lists titles and acronyms 
commonly used to identify most of the JPAs. 

Format/Content Types 

Table 1 classifies 37 JPA techniques according to (l) basic format or 
content types (modified from Post, Price, & Diffley, 1976), (2) personnel 
qualification requirements for using each technique, (3) the type of hard- 
ware system for which each technique is designed, and (4) general demonstrated 
feasibility. Middleton, Guilliams, and Weber (1977) criticized the Special 
Case formats, pointing out that they can generally be subsumed under the 
major categories for troubleshooting and nontroubleshooting aids. With the 
exception of the Quick Fix procedure, the analyses performed for this task 
confirmed that finding. However, the Special Case categories are useful in 
ttiat they suggest unique requirements (e.g., hazardous conditions, time, one- 
trial learning) that could be major considerations in some systems. Also 
available are specifications that deal independently with certain of the 
Special Case aids (i.e., maintenance toquirement cards and maintenance de- 
pendency charts) and that give them a uniquenftss for both performance and 
cost considerations that could be lost by attempting too broad a classifi- 
cation. 
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The discussion of Post et al. (1976) on the Special Case aids appears 
adequate for describing the unique features of Troubleshooting, Nontrouble- 
shooting, and Special Case JPAs. One aid type that they did not discuss 
but that is listed in Table 1 is that of highly complex integrated systems, 
such as CMGj CONSD, FOMM, SIMM (listed in the glossary with FOMM) , MDC, 
and Work Package. The CMG and CONSD were. designed only for experienced 
personnel, while the FOMM, MDC, and Work Package are aimed at both fully 
and lesser-experienced personnel. Of this set, only the SIMM and MDC are 
supported by published experimental results, and those results have not 
been overly favorable toward the JPA. In some cases, they have even been 
negative (Rowan, 1973; Shriver & Hart, 1975). 

The FPJPA, exemplified by Figure 1 (Westinghouse, 1973), is unique 
among existing JPA format or content types. From the early Army studies 
until the present time, it has had the most consistently positive experi- 
mental results. Because of this, the FPJPA' has been mistakenly regarded by 
some as being the only type of JPA. It forms the basis for the JTM/JTG 
format in the Army iTD'f program and has been the preferred format of the 
Air Force in its attempts to implement job aids on the C-141 program; both 
services have prepared specifications for procuring both nontroubleshooting 
and troubleshooting FPJPAs. The Navy has not done so, but it does have the 
NAVAIR Work Package specifications and the HAVAIR/HAVSEA writer guidelines, 
which include many of the features of the FPJPA for nontroubleshooting 
operation and maintenance tasks. The Work Package concept, which is clas- 
sified in Table 1 as a partially proceduralized aid, also provides for 
a troubleshooting decision logic tree. 

Only the FPJPA types of JPAs (i.e., Air Force FPJPA, AMSAS, C-1A1 Aids, 
JOBTRAIN, OPTIMUM PIC/WOKD, and PIMO) have demonstrated positive results 
with the novice technician (acceptable aptitude but inexperienced). Posi- 
tive results have also been claimed for the Work Package concept on a 
demonstration with organization- and intermediate- level maintenance on the 
APQ-125 radar (Horn, Note 2), but the results have not been published. 
The positive results of the MDC (Krohn, Clayton, & Dunn, 1971) with in- 
experienced personnel were confounded by the lack bf an experimental con- 
trol, and therefore are not conclusive. 

The FPJPA, FOMM, and Work Package have all been fully developed for 
electrical, mechanical, and electromechanical systems, but the FOMM generally 
is not considered adequate for digital electronics or mechanical systems. 
As shown by Table 1, no single JPA system concept includes all levels of 
proceduralization . The only true attempt to combine two or mora levels of 
proceduralization was the AAT, a NAVAIR hybrid aid for troubleshooting that 
provided two separate routes to successful fault isolation. This technique 
introduced special cues designed (1) to allow the technician to move from 
one route to the other and (2) to suggest why certain tests were being 
wade. The latter feature was to encourage greater incidental learning of 
higher order troubleshooting skills. 



CAE-RA GUUOuT 

1. Observe oscilloscope screen and depress camera SNUnLft bar 17). 
liote Wit oscilloscope trace (2) moves across screen 4-1/2 to 
5-1/2 divisions *3) before dropping to tue lower level 11). 

t. If repeat of test is necessary, depress RESET button (5) tnen 
depress SHUTTER fcar (7), 

Does si"3ep drop oetween limits (3)? 
j T *0 j Go to page 1-26. 



3. Set camera SHUTTER SPEED fcnoi? (6) to 2". 

4. Set oscilloscope TIM£/CM fcnoo (4) to ,1 SEC. Depress oscil- 
loscope RESET outton (5). RESET button snould Tight. If not t 
depress RESET out ton (5) again. * 

5. Ooserve oscilloscope screen very closely and depress camera 
SHUTTER oar (7), ilote tnat oscilloscope trace i2) moves across 
screen 4-1/2 to 5-1/2 divisions £3) before dropping to tne 
lower level (1). 

6. If repeat of test is necessary, depress RESET button (5) t>ien 
depress SHUTTER bar (7). 



Does sweep drop between limits (3)? 
| vEV> Proceed to next page. 

1 hq% Go to pa^e 1-87. 




Figure 1. Example of PPJPA format/content, 
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The discussion of Post et al. (1976) on the Special Case aids appears 
adequate for describing the unique features of Troubleshooting, Nontrouble- 
shooting, and Special Case JPAs. One aid type that they did not discuss 
but that is listed in Table 1 is that of highly complex integrated systems, 
such as CMG, CONSD, FOMM, SIMM (listed in the glossary with FOMM), MDC, 
and Work Package. The CMG and CONSD were designed only for experienced 
personnel, while the FOMM, MDC, and Work Package are aimed at both fully 
and lesser -experienced personnel. Of this set, only the SIMM and MDC are 
supported by published experimental results, and those results have not 
been overly favorable toward the JPA. In some cases, they have even been 
negative (Rowan, 1973; Shriver & Hart, 1975). 

The FPJPA, exemplified by Figure 1 (Westinghouse, 1973), is unique 
among existing JPA format or content types. From the early Army studies 
until the present time, it has had the most consistently positive experi- 
mental results. Because of this, the FPJPA has been mistakenly regarded by 
some as. being the only type of JPA. It forms the basis for the JTM/JTG 
format in the Army 1TDT program and has been the preferred roraat of the 
Air Force in its attempts to implement job aids on the C~141 program; both 
services have prepared specifications for procuring both nontroubleshooting 
and troubleshooting FPJPAs. The Navy has net done so, but it does have the 
NAVA1R Work Package specifications and the HAVAHt/NAVSEA writer guidelines, 
which include many of the features of the FPJPA for nontroubleshooting 
operation and maintenance tasks. The Work Package concept, which is clas- 
sified in Table 1 as a partially proceduralized aid, also provides for 
a troubleshooting decision logic tree. 

Only the FPJPA types of JPAs (i.e., Air Force FPJPA, AMSAS, C-141 Aids, 
JOBTRAIW, OPTIMUM PIC/WORD, and PIMO) have demonstrated positive results 
with the novice technician (acceptable aptitude but inexperienced). Posi- 
tive results have also been claimed for the Work Package concept on a 
demonstration with organization- and intermediate* level maintenance on the 
APQ-126 radar (Horn, Note 2), but the results have not been published. 
The positive results of the MDC (Krohn, Clayton, & Dunn, 1971) with in- 
experienced personnel were confounded by the lack of adequate experimental 
control, and therefore are not conclusive. 

The FPJPA, FOMM, and Work Package have all been fully developed for 
electrical, mechanical, and electromechanical systems, but the FOMM generally 
is not considered adequate for digital electronics or mechanical systems* 
As shown by Table 1, no single JPA system concept includes all levels of 
proceduralization. The only true attempt to combine two or more levels of 
procedurally at ion was the AAT, a HAVAIR hybrid aid for troubleshooting that 
provided two separate routes to successful fault isolation. This technique 
introduced special cues designed (l) to allow the technician to move from 
one route to the other and (2) to suggest why certain tests were being 
made. The latter feature was to encourage greater incidental learning of 
higher order troubleshooting skills. 




Several techniques <e*g., ATOMS, AVIS, FOMM, MAINTRAIN) are sufficiently 
flexible to be classified as both simple logic deductive and system descrip- 
tion types. Some (e.g. , JOBTRAIN, Work Package) even provide for three 
levels of troubleshooting flexibility, but none were found that mixed 
FPJPA formats with the other oasic formats. This is probably due to two 
factors: First s FPJPA is a unique form of aiding that relies on a complete 
front-end analysis and fully illustrated procedures for each task being 
aided while making no assumption for deductive reasoning abilities or 
formal training. The other aid forms assume at least a midlevel technical 
background achieved through training or experience. The second factor is 
that, in both the Army and the Air Force, the FPJPA has been promoted 
as a substitute for the conventional technical order or .technical manual, 
not as an adjunct or supplement to it. It was thought that no program 
manager would readily approve expenditures for two parallel aiding systems 
on the same equipment. It has now become evident that no existing aid can 
meet all the objectives of cost, performance, and manpower of a particular 
organization at the command level. There is not even an easy division 
where FPJPA, FOMM, or the Work Package can be used that will be consistent 
over time across major groups of programs. There are instances where all 
13 aid forms may be necessary on a single system, while on others only 
one may be required. 

Display Media Types 

Table 2 classifies 47 selected JPA techniques according to the 7 basic 
media display types. The differences in types of display and special 
physical features are much easier to discern than are the format/content 
types, but it is probably in this area that the greatest confusion exists 
regarding the relative advantages and disadvantages of performance aiding. 
There is little evidence to support the contention that performance can be 
further improved beyond that obtainable with hard-copy print simply by the 
use of other types of display media. However, there are exceptions, some 
quite impressive, that will be pointed out in the following discussion. 

Hard-copy Print Displays 

In hard-copy print, the major physical feature of interest is 
size. Printed performance aids can vary from hand-Weld cards, checklists, 
and booklets to large wall charts or manuals with table size foldouts. 
For presenting operator and maintenance instructions, PIMO has short, 
step-by-step instructions and facing illustrations bound in a pocket-size 
booklet; the MDC and FLAP formats use oversize pages so that complete task 
information can be presented on a single page. 

There are many other uses for harJ-copy print JPAs than just opera- 
tor and maintenance procedures, particularly in the smaller formats. For 
example, Chalupsky and Kopf (1967) described hard-copy devices used as aids 
in product inspection tasks, as management aids, as nomographs, and as 
conversion charts. . Conversion charts in particular have been especially 
useful in patient care for such things as making conversions from the 
apothecary to the metric system. Hooprich and Steinemann (1965) found that 
a wallet-size electrical measurement chart raised measurement conversion 
performance of electronics technicians with little or no training to a level 
equal to that of experienced technicians. 
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Table 2 

Display Media Types, Special Physical Features, 
and Feasibility Factors Applicable to Various JPA Systems and Techniques 



Display Media Type 
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There have been other physical features of interest in the print 
format—for example, tab size (Rees fit Kama, 1959); tabular displays 
(Wright fit Fox, 1972); and special page coatings to resist grease, dirt, 
and darapness-*-but design questions apart from size have more frequently 
concerned the desirability of embellishments. Usually, attempts to em- 
bellish a printed a id employ color or cartoons. If such changes (e.g., 
color coding for search aiding, organization improvement, or reduction 
in reading requirement) enhance comprehension, then they may be desirable, 
particularly with low-aptitude groups; but when used strictly for their 
attention-getting properties, they do not usually increase acceptance or 
improve performance. AVIS experimented with a female voice as an audio 
embellishment but, rather than improving performance, it proved to be & 
distraction and tended to reduce confidence in the message. 

Audiovisual Displays 

Audiovisual Ca/V) devices have been expected to play a much larger 
role in performance aiding, but military studies specifically designed to 
compare such devices, such as Audiscanpn the Air Force 0141 aircraft 
(Goff, Schlesinger, £ Parlog, 1969) arri AVIS, on the NIKE missile system 
(U. S. Army Test and Evaluation Directorate, 1964), have shown performance 
aided by A/Vs to be poorer than perfc :mance aided by visuals alone. 

The implementation of A/V d&pices for performance aiding has not 
met with much success in the civilian sector either. Chalupsky and Kopf 
(1967), in their survey of 12 firms, found that the installation of A/V 
devices increased rapidly about 1960 but by 1963, few remained In full- 
time service as on-line aids. Travers (1964) explained that an A/V device 
IS most likely to be of value when the rate of information input is very 
slow, so that the user can switch from one mode to the other without be- 
coming overloaded with information. 

One particularly novel A/V device (ELECTkOCULAR) , developed by the 
Office of Naval Research in the early 1960s, was a lightweight, head-worn 
unit that contained a 35mm filmstrip and an audio tape carrying operation 
or maintenance instructions (Brown, 1964). The unit's optical system 
allowed the user to see both the instructions and the work environment. 
The advantage of this device was not that it increased comprehension, 
due to the combined sensory modality presentation, but rather that it 
allowed greater flexibility. The device was most useful in presenting 
maintenance instructions where there was insufficient space for larger 
displays oc where the operator or maintainer needed to use both hands 
or to be mobile. 

Audio or Visual Displays 

Audio systems and visual displays (in addition to hardcopy print) 
have been more successful as separate aiding devices than have the coifr- 
bined A/V units. 

Audio Systems . JPAs with programmed audio instructions on tap* 
have been reported by Chalupsky and Kopf (1967) to be highly successful 
for cable harness or cable board wiring in the electronics industry; how- 
ever, only one device of those surveyed (SADIE) was a purely audio JPA for 
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a military application. This device consisted of a lO^lb. (4.53 kg.) 
specially designed, two-track magnetic tape player and a handHield controller. 
One track contained ft series of detailed troubleshooting instructions while 
the other track contained the troubleshooting sequence logic in digital form. 
A "yes" button advanced the tape to the next instruction; a "no" button, to 
another step in the sequence. Although impressive claims were made for the 
device, it is not known to be in use and no experimental results have been 
reported in the literature. 

Perhaps the best use of the audio channel has been in real-time 
voice communication, ranging from a supervisor instructing a worker directly 
to the use of a voice link connecting one station to another. Examples of 
the latter are the McDonnell Douglas FEFI-TAFI and the Quick ^ix proce* 
dures, in which the pilot or an in-flight maintenance crew member alerts 
a ground station to a maintenance need by means of a coded signal. 

Visual Displays . Photos, picture guides, slides, CRTs, 3-D models, 
and plastic overlays already have been employed extensively as JPAs. For 
example, the implosion technique, which used slides, allowed the user to 
form perceptual blueprints of how mechanical parts fit together or how elec- 
trical networks connect, but it has only been applied to training. The 
SHOCK ACTION technique provided picture guides for tank crewmen to use as 
on-the-job performance aids. The MMS system employs a computerized CRT for 
displaying and manipulating the Maintenance Dependency Chart (MDC) . TRACE 
uses a plastic overlay on circuit boards to show test points, current flow, 
I and test data to simplify the testing of transitor radios. Overlays have 

also been used in production facilities for the assembly of printed circuit 
cards. In one case, the arrangement of component bins was integrated with 
overlays to eliminate assemblers* decisions in selecting or locating com- 
ponents to be attached to the printed card. 

Microform Systems 

- Microform is often considered an information transfer medium that 
can be substituted for hardcopy print or A/V devices. Actually, however, 
the advantages of microform systems alone are not really competitive with 
such devices on the same aiding factors. While print and A/V devices are 
primarily directed toward aiding information comprehension, microform systems 
are designed primarily to ease information access. All such systems used as 
job aids are storage and retrieval devices that provide a more convenient 
means of accessing large information sources at the work site. Decisions 
to use microform systems for on-the-job aiding are based not on direct user 
requirements, but on such criteria as reduced storage requirements or re* 
duced distribution and update costs. 

No studies were found that directly compared microform display 
systems to both A/V and print displays for relative advantages in job-oriented 
training or job performance. However, a few experimental studies compared 
microform systems against the print mode on training success criteria. The 
most recent of these (Rizzo, 1977) compared the efficiency of microfiche 
versus traditional paper copy for training Navy personnel on the Basic 
Electricity and Electronics (BE&E) curriculum. That experiment measured 
(1) the time it toojt individuals in three aptitude groups to complete 14 
individualized, self-paced modules and (2) errors on comprehensive examina- 
tion tests. An important finding was that the lower aptitude group was 
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adversely affected (more than 25% greater training time) by the microfiche 
medium. These results are similar to those of an earlier Air Force study 
(Grausnick & Kottenstette, 1971) that found no advantages with microfilm 
for any aptitude group and definite disadvantages for lower-aptitude per- 
sonnel . 

Novel Displays and Devices 

While the results with audiovisual and microform systems have been 
disappointing, some nonpaper aiding devices appear to offer considerable 
advantages over static paper displays. Examples are (1) devices such as 
holography and three-dimensional models, and (2) interactive devices such 
as hand-held computers and slide rules. Frey (1976) found the holographic 
medium to be as good as or superior to line drawings and photographs on 
assembly, discrepancy identification, and information extraction tasks. 
The production of good quality holograms is still an art, however, and no 
attempt has been made to produce them for use in schools or in the field. 

Interactive devices can almost be considered as a distinct class 
in aiding technology. These devices, which can include slide rules, hand- 
held computers, and interactive CRT displays, are much more dependent on 
the state-of-the-art in information handling than are other devices, Which 
tend to present one-way, relatively fixed instructions. Chalupsky and 
Kopf (1967) classified these as task modifying aids because they allow the 
* user to achieve a desired result without performing all of the steps that 

normally would be necessary. Such devices are contrasted with task supporting 
aids such as procedural ized instructions, which guide task performance but 
result in the same behaviors that would occur without an aid. 

Recent work of interest in this area is that of Burkett and Kruse 
(Note 3) on card-programmable and chip-programmable pocket calculators. 
They claim to have discovered an incidental learning process they call 
"Thinking Process Transmutation." In this process, the user is guided 
incidentally into reorganizing his own thinking processes to fit the cal- 
culator's logical processes. Thus, the user seems to develop a new capa- 
bility to organize thoughts, to define necessary steps, and to develop the 
most efficient procedures for problem solution. The potential of such appli- 
cations seems enormous in equipment fault isolation or in operator analyses 
in command-and-control decision making, but only one study has been found 
that actually applies the concept to job aiding (Rigney & Towne, 1977). 

One unique method, called the Concentrated Odor Sensing Technique 
(COST), uses the sense of smell. Developed by the Illinois Institute of 
Technology, the technique employs up to 128 different odors concentrated 
and encapsulated in tiny seeds designed to be melted by heat, cracked open 
by pressure, or dissolved by a liquid, thereby releasing the odor. As an 
aiding device, the capsules could be used on electrical circuits, motors, 
pumps, hydraulic lines, or bearings to indicate trouble areas. 

Applied Training 

Of the systems surveyed, several aid on-the-job performance through 
applied training techniques. The importance of discussing such systems 
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here 13 that these techniques, to some degree, have been designed for imple- 
mentation in conjunction with job aids. Training designed in this manner 
tends to be most compatible with job aiding and has many of the same goals, 
including (1) the reduction of total class and Instructor time, (2) imparting 
only those skills necessary for performing a job, (3) an emphasis on proficiency 
rather than knowledge tests, and (4) greater "hands-on 11 experience* 

The older Army systems (e.g., FORECAST, JOBTRAIN, MAINTRAIN) ware 
specifically designed around a combination of training and job aids that 
would require less training time than conventional courses. Some of the 
Navy-developed maintenance training concepts (e.g., TASKTEAOH, MITIPAC, 
ISO) s£em to consider interactive CAI and maintenance simulation to be the 
heart of Improvements in job performance, with prescriptive aids and tech* 
nlcal documentation being incidental to the training process. Such tech- 
niques need to be placed on an equal footing with aiding to fully achieve 
the advantages of aiding technology. 

JOBTRAIN's training objectives Included "theory, 11 but such Instruction 
was Introduced only as the student needed it to solve practical problems. 
In FORECAST, the cue-response or behavioral type of task analysis was Intro* 
duced as the basis for both training and job aiding. In FORECAST and JOS- 
TRAIN the emphasis on theory also became more systems or functionally oriented 
than in the past. The shift in orientation toward logical approaches to 
troubleshooting greatly reduced the need for knowledge of the fine points 
of electronics theory. Theoretically, this trade-off (preliminary analysis 
with documented logic versus knowledge training) could be pursued to its 
natural limit; all information in the book for use directly on the job 
and no information presented beforehand for instilling knowledge in the 
head . It has been recognized for some time, however, that certain informa- 
tion necessary for job performance is better assigned to the head than to 
the book (Chalupsky & Kopf, 1967; Chensoff, 1973). 

Although convenient for JFA/training task assignments, the distinction 
of head versus book does not necessarily parallel that of training versus 
aiding. Contractors on the Army ITDT program are finding that many of the 
knowledges and skills needed to do productive work with available job aids 
can be acquired through the book (in an aiding format). One effort, for 
example, reduced the training requirement for tank maintenance tasks from 
200 to 18 after finding that the majority of training tasks could be learned 
by simple procedural ized training aids that are indistinguishable from job 
aids. Many of the remaining skills seem to be learned incidentally while 
performing with job aids. 

There appears to be less and less reason for the separation of training 
and aiding in the initial stages of enhancing job knowledges and skills. 
This is not to say that job aids can totally replace training. Teel and Chaney 
(Note 4), in one of the few experiments comparing training to aiding, showed 
that combining the two produced almost double the performance improvement 
of either method used alone. Furthermore, as Klesch (1977) pointed out with 
job-oriented training programs such as learner centered Instruction (LCI) 
and job aiding programs sue)) as PIMO, all of the problems of personnel careers 
(e.g., skills development, cross training, retention) must be considered if 
either is to be successful. 
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For example, neither applied training nor fully proceduralized job aids 
adequately address the long-term personnel problem of retaining and building 
a career force in any of the services. This then leads to the need to 
consider training and aiding as changing in relative usefulness over an 
individual's career. 

Peripheral Test/Diagnostic Systems 

Some JPA systems combine automatic testing diagnostic features with 
further diagnostic and repair instructions for maintenance personnel to 
use in correcting malfunctions. Such systems usually cover electronic 
equipment and, to a lesser extent, hydraulic, pneumatic, and other flow 
systems. Three basic types of aiding systems can be classified under this 
category: (1) built-in, computer aided tests, (2) status displays, and 
(3) fault-probability displays. 

The first type is more commonly known as ATE (automatic test equipment), 
but to be classified as a JPA, a system must be capable of (1) diagnosing 
the problem down to the level of repair and (2) providing the means for 
aiding the repair itself. Examples in the glossary are the Westing house 
ADHIRE/MIRAID and the Vought Aeronautics CATA, which have built-in equip- 
ment sensors that provide inputs directly to a computer for first^level 
diagnosis. The test equipment then provides follow-up instructions for 
the maintenance technician. Other systems, such as VAST and VATE, are 
even more automatic in that they use computerized fault isolation to indi* 
cate which unit needed to, be replaced. In such systems an aid is most use- 
ful for operation and maintenance of the ATE -or as a backup when the ATE 
is malfunctioning. 

The second type, "status displays," either key lists of displays to 
troubleshooting instructions or include a decision logic table for com* 
pleting the aiding function (Post et al., 1976). A decision logic table 
may be preferable where a large number of displays are present since it 
can provide patterns of display readings that more readily guide the tech- 
nician to the trouble areas. A good example of this type of aiding system 
is the FEFI-TAFI system on DC-10 aircraft. The system controls and dis* 
plays that normally are provided for commanding and monitoring the aircraft 
or the operation were found extremely useful in indicating system failures. 
Earlier studies by McDonnell Douglas had found that, typically (since several 
indicators and conditions sensed by the flight crew are associated with 
specific subsystems), malfunctions in different components produce relatively 
unique patterns of normal and abnormal indications that can be considered 
signatures of particular subsystem failures. Figure 2 (Burrows, Willets, 
& Miles, 1970) illustrates a typical flight engineer's fault isolation (FEFI) 
decision logic table of indicator patterns. 

The final type of aiding system in this general category is that of 
probability-of-fault systems. Usually, these attempt tc use historical 
data of system usage to improve reliability prediction. The aid allows 
a technician to go directly to a fault, based on major trouble symptoms 
and reliability prediction data. 
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Figure 2. Typical FEFI decision logic table of indicator patterns. 



Peripheral test/diagnostic systems are most useful where repairs need 
to be done rapidly, as on an aircraft carrier flight deck where a quick 
turnaround between landing and launching is needed. Of the systems sampled,* 
those that have attempted to facilitate such rapid repairs are the Grumman 
Aircraft RAPIDS and the REP0M/M1RM system sponsored by the Haval Ordnance 
Laboratory* Although very helpful in rapid fault isolation, such systems 
have not been widely implemented because of the high cost of obtaining, 
maintaining, and analyzing the data necessary to continuously update the 
maintenance instructions* 

Delivery Systems 

: Several types of systems that are classified as aiding systems are 
really process systems for developing and supporting the aid itself* 
Usually, such systems provide and update technical information to opera* 
tors and maintainers and, as such, are outgrowths of the various technical 
manuals (also referred to as technical data, documentation, publications, 
and orders) systems for commercial and military applications* In this class 
of systems there are several stages in the process that could contribute 
greatly to the quality of the final aiding product. Kany of the JPA systems 
identified in the glossary are subsystems of complete job aid delivery systems 
TRUMP, for example, is concerned primarily with the rewrite and update of 
technical publications, while MIARS (listed in the glossary with TRUMP) 
is a microform system directed toward information storage, distribution, 
and retrieval. 

The two systems of most interest in this discussion are the Navy NTIP 
* system and the Army ITDT program* NTIP is concerned with each phase of the 
delivery process but places considerable emphasis on hardware systems for 
technical information production, handling, and display* The ITDT is 
primarily concerned with supporting both the publication and training ;ieeds 
of weapon systems and equipment, with heavy emphasis on a front-end analysis* 

The NTIP has identified eight necessary stages in JPA delivery (Hughes, 
1977a, 1977b) — user^data match requirements, data acquisition, content genera- 
tion, content capture, content replication, distribution, update, and feed* 
back—and has identified some research needs in each stage (Figure 3). 

User-data Match Requirements 

The user-data match stage, in specifying the appropriate aid formats, 
content, and media for any particular project, considers (1) personnel charac- 
teristics, (2) the results of the analysis of the equipment or system and 
task, and (3) environmental constraints* The primary media considered under 
this system are paper manuals, A/V devices, microform, and video disc devices* 
This stage is most similar to (but not as rigorous as) the Amy's front* 
end analysis. It is probably at this stage that the various JPA selection 
algorithms should be used. Determining the level of detail required and the 
quality of the task analysis (front-end analysis) to be performed are the 
most critical decisions to be made at this stage. 
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Data Acquisition 



Data acquisition includes the specifications, management, policies 
and procedures, and data acquisition organizations involved in acquiring 
the JPAt Xn addition to specifying front~end analyses and media, attempts 
should be made to introduce quality assurance provisions to ensure that the 
delivered JPA is complete, accurate, comprehensible, and usable* The problem 
of accuracy, particularly with troubleshooting JPAs, appears to be the most 
difficult to solve and the least likely to be improved by increased rigidity 
in the specifications. Braid (1977) suggested that this problem can only be 
solved through more intelligent use of front-end analysis and feedback. 

Content Generation 

The content generation stage of the JPA process includes actually 
developing the content to be displayed with a JPA* Eight steps are involved; 
(l) developing or obtaining and accessing the engineering or manufacturing 
data base, (2) developing planning documents, (3) making interface assign* 
ments with training, (A) performing maintenance engineering, provisioning, 
and design engineering, (5) conducting the front-end analysis, (6) writing 
and illustrating the JPA, (7) validating the JPA, and (8) compiling program 
historical data. Regardless of the final user medium, the end product of 
the content generation stage will almost always be a draft JPA on paper. 

Content Capture . 

The content capture function transforms the draft JPA into a form 
and format suitable for replication. It combines the functions of text pro* 
cessing (usually by typewriter), graphics processing (manually prepared or 
computer-assisted), and production, with the output being a reproducible 
copy in a predetermined medium. This entire function can be automated with 
state-of-the-art equipment. TRUMP is one of the most sophisticated military 
systems for this function. It uses optical character recognition (OCR) 
to scan printed pages, with graphic inputs processed in photographic form* 
The system can also receive alphanumerics, edit, and make data correction 
inputs through interactive terminals. Outputs can be either microform or 
reproducible copy. 

Future systems are expected to store all information digitally. 
The methods for digital storage that have been considered include magnetic 
tape, magnetic disc, magnetic drum, optical, semiconductor, hologram, and 
video disc. The video disc and hologram appear to offer the most promise 
for mass^storage techniques, but neither is currently suitable for applica- 
tion in JPA development systems. 

Content Replication 

Content replication involves printing and copying technology. 
While the technology for copying has made dramatic progress in the past 
years, printing technology as can be applied tc JPAs has not changed appre- 
ciably for many years. The printing process includes several steps (i.e., 
imaging, stripping, making the printing plate, printing, and binding), 
but it is only in the binding process that automation is more fully developed 
for JPAs. It is unlikely that JPA printing technology for delivery systems 




will change much in the next few years because of the high capital invest- 
ment in machines now being used and because JPAs are usually short-run but 
relatively .large, especially when used as maintenance information aids. 

Improvements in copying offer more possibilities to JPAs, par- 
ticularly in the area of on-site, on-demand, quick-printout capability 
and in the reproduction of color originals. In addition to present-day 
microform copies, it is expected that technical information, Including 
graphics, soon could be delivered in digital form at the work site, where 
it could then be replicated on CRT displays or produced as printed copy. 

Distribution 

The process of distributing JPAs for complex military weapon systems 
implies much more than a stockpile of aids that can be delivered to user 
activities. The distribution system must provide for (1) initial distri- 
bution to JPA users, (2) automatic distribution of future issties, changes, 
and corrections, and (3) special one-time requests. In military, systems, 
configuration changes are common due to continued efforts to improve equip- 
ment operability and reliability • \ Also, equipment maintenance often turns 
out to be somewhat different than originally planned (e.g., planned dependence 
on automatic test equipment that becomes unreliable), JPAs require accuracy 
and must reflect the actual equipment being operated or maintained if they 
are to function properly. Assigning a configuration or identification 
number to the JPA for administrative control, and tracking the JPA with data 
management techniques throughout its life cycle, are the major means of, 
ensuring such accuracy and fidelity. At present, no system exists outside 
of the technical manual or technical documentation fields with the means 
or responsibility for performing this critical function. The distribution 
process also must .maintain (1) adequate supplies for rapid response to re- 
quests and (2) anarchive for historical reference data and supply stock 
replenishment. 

Feedback 

The feedback link from the JPA users to the distribution and change 
systems is critical to JPA effectiveness. In fact, full utilization of this 
link may be the only way that troubleshooting JPAs (of the proceduralized 
variety) can be made to satisfactorily reflect the troubleshooting routines 
needed on any particular system or equipment. This is a particularly weak* 
link in current technical manual systems — it is slow, user participation 
is low, and too little direct communication occurs between the user 'and the 
data manager. The situation is not likely to improve until the user is con- 
vinced that his efforts in detecting and reporting inaccuracies in the JPA 
are considered seriously and expeditiously. 

Update 

The update subsystem connects the configuration control and feedback 
system to the distribution process. It has content capture and content 
generation functions (as already discussed) for original materials, but 
also must consider methods for changing original material and for inserting 
new material* Inherent in th£ update process is a methodology for deter- 
mining the effects of a change in one JPA upon other data and logistic ele- 
ments. When a system is in production, there should be provisions for 
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coordinating all logistic! elements. The coordination among elements for 
out-of-production equipment tends to be under the control of Navy facili- 
ties and Is less formalized, but the problems of configuration control 
are considerably reduced by the time an equipment reaches this stage. 
Here again, technical data\ systems have made bona fide attempts to control 
and manage this process, but it still is done less efficiently than would 
be required for a workable JPA system. 
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JPA ALGORITHM DEVELOPMENT FACTORS 



In the development of a JPA algorithm* four general areas of JPA tech-* 
nology must be considered: (1) factors important in selecting the baaic 
format or content type of JPA, (2) requirements for special (i.e. , nonprint) 
media or special physical features of the JPA, (3) the relationships between 
the JPA, training* and job design, and (4) JPA test and evaluation tech- . 
nology. 

JPA Selection Factors 

In designing JPA systems to meet the personnel, training, and maintenance 
support requirements of a hardware system* the following variables are 
primary considerations in "selecting the basic JPA system type: 

v 1. Personnel aptitude. 

2. Personnel job experience. 

3. Type of task being aided. 

4. Complexity of the task. 

5. Complexity of the equipment. 

6. Type of equipment. 

7. Degree of proceduralization. 

Various combinationa of these variables have been investigated for the rela- 
tive effects of one system over another on performance, training, or job 
design. 

Personnel Aptitude 

Rowan (1973) stated that the JPA and training literature is replete 
with studies indicating that lower-aptitude personnel can carry out non- 
troubleshooting maintenance tasks with minimal training using properly pre- 
pared job performance aids. Because of (1) decreasing aptitude levels among 
new enlistees and (2) a shrinking manpower pool at the high-aptitude levels, 
JPAs are being looked to as a way of better utilizing lower-aptitude per- 
sonnel. 

Overall, however, the results are not as convincing aa Rowan haa 
r suggested* While numerous studies report the advantages of aiding inex- 
perienced personnel, few directly address the problems of lower-apt ituda 
personnel. Available data do tend to support Rowan* a interpretation, but 
even the best studies base their conclusiona on a finding of no atatlstical 
differences. Moreover, in studies that found no differences among aptitude 
levele, thia finding was usually limited to non troubleshooting teaks (Elliott, 
1967; Hooprich & Steineaann, 1965). Elliott, for example, in cohering 
procedural ized job guide performance for 20 high echool aeniore— 10 high 
eptitude (80*95 on the AQB Electronic Index) and 10 medium aptitude (50-65 
on the AQE)— on troubleshooting end repeir teaks, found that epti^sda had 
no effect on errors in maintenance repeirs or on the time required for 
troubleshooting. There was a etetistically aignificent difference, how- 
ever, in errore made on troubleehooting et both the syetem nodule and defec- 
tive components levels; that is, high-eptitude performance wee better than 
low*aptitude performance. 
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There have also been reports of successful non troubleshooting per- 
formance by personnel considered to have lower aptitude levels, although 
the studies were not specifically designed to test the aptitude factors. 
Horn (1972), for example, found lower-aptitude mechanical ratings able to 
do electrical ratings* nontroubleshooting jobs with FPJPAs. Greater per- 
formance from lower-reading-ability personnel may also be inferred from the 
successful performance of maintenance tasks by non-English speaking Vietnamese 
on the American UH-1H helicopter. 

Only one study has suggested that lower-aptitude personnel can 
also do troubleshooting accurately with the FPJPAs Theisen and Fishburne 
{in press), in a field^study demonstration with 41 Navy technicians whose 
selection test scores were below the cutoff for ,f A f1 school eligibility, 
concluded that the subjects were able to satisfactorily isolate and repair 
malfunctions in the AN*AQA*7 Sonar Signal Processing System with the use of 
the FPJPA. Unfortunately, the results of this study were greatly weakened 
x * by the lack of adequate control data. 

The consensus of researchers in the field is that aptitude trade* 
off has not been adequately studied. However, most are confident that, 
whatever the level of performance of the lower-aptitude personnel, increased 
training and the availability of JPAs can bring even very low-aptitude 
personnel (i.e., fifth-grade reading level or below) to acceptable perfor- 
mance levels in nontroubleshooting operator and maintenance tasks. 

Job Experience 

Unlike aptitude, the effects of experience are adequately con- 
sidered in most JPA experimental research. Rowan (1973), in summarizing 
the literature, noted that all FPJPA experiments and field tests showed 
that inexperienced technicians performed better with aiding than with con- 
ventional documentation. This has been generally true for troubleshooting 
as well as nontroubleshooting tasks, although acceptable levels on complex 
troubleshooting tasks are more difficult to achieve and FPJPAs have failed 
completely in some instances (Klesch, 1977). 

Some JPA techniques other than FPJPA have succeeded in raising the 
performance level of inexperienced technicians in troubleshooting} for 
example, the Augmented Action Tree (Horn, 1973), Logic Tree Troubleshooting, 
Aids (Potter & Thomas, 1976), and the Work Package Concept (Horn, Note 2). 
All, however, have included some form of decision logic tree. One non~ 
FPJPA technique that has received considerable attention is the MDC Studies 
of the use of this technique by inexperienced personnel have yielded mixed 
results (Krohn, 1971; Horn, 1973). 

Also, although all experience levels generally express more favor- 
able attitudes toward proceduralized JPA techniques (Goff et al., 1969; 
Potter & Thomas, 1976), well-prepared conventional documentation in the 
hands of experienced personnel can result in better performance on some 
types of troubleshooting tasks (Potter & Thomas, 1976). Highly procedura- 
lized techniques alone probably cannot support all of the troubleshooting 
demands of complex weapon systems (Klesch, 1977; Booher, 1977b). 
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Type of Task 



For determining the major differential effects of aiding techniques, 
it is now clear that the type of maintenance task being aided can be clas- 
sified as either troubleshooting or non trouble shoo ting. 

It has long been known that mechanics can become relatively skilled 
on nontroubleshooting tasks after 6 months of field experience, whereas 
troubleshooting skills continue to improve throughout their careers (Baldwin, 
Mater, Vineberg, & Whipple, 1957). With job aids, inexperienced and lower- 
aptitude personnel can be proficient on nontroubleshooting tasks almost 
immediately, essentially eliminating the first 6 months of warm-up. Con- 
siderably more time is required, even with job aids, to become proficient 
in troubleshooting skills, but proficiency can be achieved much earlier ift 
the technician's career with aiding. 

Procedural ized job aids are less expensive to develop for nontrouble- 
shooting tasks than for troubleshooting tasks because the former require no 
functional analysis (Shriver & Hart, 1975) and are much easier to validate: 
for accuracy (Johnson, 1977). According to Middleton et al. (1977), non- 
troubleshooting aids are also relatively easy to tailor to the degree of 
proceduralization needed to meet individual maintenance task requirements. 
The aiding of troubleshooting tasks, on the other hand, generally requires 
a functional analysis, can be difficult to validate for accuracy, and inter- 
acts considerably with variables such as technician experience, equipment 
complexity, and task complexity. 

Complexity of Task and Equipment Factors 

Studies have generally attempted to deal with the complexity of 
tasks and with the type and complexity^of equipment by sampling across 
several difficulty levels and types rather than considering them to be 
major independent variables. Clear differences due to the these factors 
have not been evident. However, some general rules of thumb seem to have 
evolved during 20 years of JPA research. First, if the task is a nontrouble* 
shooting task, then task complexity and the type and complexity of equip* 
ment have little bearing on the decision concerning the type of JPA. Second, 
some types of tasks, because of their special complexity (e.g., difficult 
alignment, adjustment tasks), might require additional training. Finally* 
if personnel of varied experience are to use the aid, then a dual-level pre- 
aentation of procedures may be useful. Generally* however, theae three 
factora do not create a great deal of difficulty once a JPA approach has 
been selected. The guidelines recommended by Middleton et al. (1977) appear 
quite useful as format selection criteria in accordance with varying task , 
complexity on nontroubleshooting tasks. 

If the task is a troubleshooting ualc, then taak and equipment com-* 
plexity, as well as the experience and training background of the personnel 
using the aid, help dictate which JPA is best. Short or excessive task 
length can greatly influence the decision to use either the FPJPA or a more 
deductive form of aiding. Other subfactors also contribute to complexity, 
such as (1) the subordination of equipment as determined by the top-down 
breakdown (Post, Price, & Diffley, 1976), and (2) the amount of special 
testing and unique requirements for deductive reasoning (Middleton et al. ■ 
1977). 
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FPJPA approaches are generally good for electrical, mechanical/ or 
electro-mechanical systems (Post & Brooks, 1970). The SIMM or FOMM format, 
on the other hand, is usually restricted to complex systems; it is generslly 
considered better for analog than digital electronics and not good at all 
for mechanical systems (Martin, 1975). However, no aiding systems were 
found that could be considered adequate for complex digital systems. 

Mechanical systems tend to be aided best by highly pictorial for- 
mats; at least that is the inference drawn from the genersl acceptance of 
existing pictorial aids. Haggard and Shock (1962) long ago demonstrated 
the effectiveness of visual aids for showing signs of malfunction and 
wear in equipment, and almost all mechanical maintenance personnel use the 
illustrated parts breakdown for performing maintenance and for ordering 
replacement parts. Several years ago the Navsl Ordnsnce Labor story and 
the Army's Human Resources Research Organization (HumRRO) , developed 
picture guides and manuals for operators and maintsiners of equipment 
such as tanks, guns, and mines. General Motors Corporation (1977) has 
developed a fault isolation and repair manual, done in a highly structured 
pictorial format, that already has sold more than 250,000 copies. 

Procedural izat ion 

In any model for JPA selection, probably the most critical factor 
regarding effects on performance, training, or job design is the degree of 
proceduralization. One of the earliest studies was reported by Atchley 
and Lehr (l964). Figures 4 and 5 (Atchley & Lehr, 1964) show the time and 
error performance advantages possible with proceduralized formats as com- 
pared to other types, job aids that produced the performance improvements 
summarized by Rowan (1973) and Shriver and Hart (1975) have generslly relied 
'heavily on proceduralization, as determined from the result of front-end 
analyses of equipment, operator, and maintenance task requirements. 
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Figure 4. Troubleshooting performance speed of Air Force basic 
trainees with three JPA format components (Atchley & 
Lehr, 1964). 
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Figure 5. Troubleshooting performance accuracy of Air Force 
basic trainees with three JPA format components 
(Atchley & Lehr, 1964), 
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Media and Design Factors 



In selecting the media and physical design of performance aids, two 
major constraints must be considered: (1) that generally Imposed by the 
hardware system Itself, and (2) that set by the work or learning environ- 
ment. 

Hard-copy print Is usually recommended because (l) It Is the least 
expensive medium, (2) It Is the easiest medium to use on the job, (3) 
performance studies have shown print to be equal to or as good as other 
media, and (4) the effect of JPAs on performance Is content-dependent, 
not medium-dependent. There may be situations, however, where systems 
support requirements dictate the use of media other than hard-copy print; 
for example, there could be distribution, storage, ^r updating problems 
with the print format that adversely affect the aid f s timeliness and 
accuracy. Regardless of the medium, any aid that has accurate, complete 
data is more essential to job performance than one without these qualities. 
There are also some situations where the task may not be presented feasibly 
with hard-copy print. For example, motion pictures could be used to describe 
complex alignments or adjustments; holography, to aid three-dimensional 
judgments; or hand-held computers, to facilitate assisting complex mathema- 
tical operations. In addition, costs may not always be on the side of hard- 
copy print; indeed, paper printing costs have risen exorbitantly in the past 
few years, making microform more and more appealing. 

Post, Price, and Piffley (1976) said that three primary system support 
requirements influence media selection that is based on varying system 
design and operating conditions: (1) access to .information, (2) recording 
medium, and (3) portrayal mode. The first, "access to information," is 
important because the task of locating information often consumes 20 per- 
cent of the maintenance technician's time. Important man-system interface 
considerations for access to the information depend on the availability 
of system status displays and automatic test equipment. Patterns of dis- 
play readings and ATE readouts keyed to appropriate job aids can help mini** 
mize the initial search time. The second requirement, "recording medium," 
describes such things as system size, number of systems, distribution pro-* 
cesses, 'and installations as constraints that may urge choices other than 
hard-copy print. Post et al. (1976) recommended that paper be used for 
information bases of up to 300 frames while the microform motie is more 
desirable beyond 1500 frames. The third requirement, "portrayal mode," 
is most critical in matching work place conditions to job aids utllizatli u* 
They .conclude that a single medium may not be equally suitable for all system 
aids or even for all aiding applications within a system. 

Hughes (l977b) presented a detailed matrix of environmental constraints 
between and within systems that dictate the physical design of hard-copy 
print, microform, and auuio or video. An ambitious attempt was made to 
recomnend design choices to the extent of page size and typography. The 
environmental constraints noted are generally human engineering considera- 
tions and include work space, illumination, dirt, wind, moisture, temperature, 
noise,, time, and hazards or safety restrictions. Table 3, a revision of the 
Hughes matrix, includes several additional techniques. For a discussion of 
the relative effectiveness of media characteristics developed around instruc- 
tional research, see Guilliford (1973). 
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Table 3 

Environ: intal Constraints for Job/ltalnlnS Media 



Environmental Constraints 



Medium and Characteristics 



Work 
Space 



Illumination Dirt 



Wind/ 
Moisture/ 
Temperatute 



Motion Noise Time Hazard/Safety 



Hard-copy Print 

Page Size 
Typogtaphy 
Page Layout 
Picture/Woi tl 

Composition 
Color 

Page Materiel 
Cover Materia* 
Cover ond Blotting 
Graphics 

Microform 

Portable Viewer 
Hand-held Viewer 
Printout C^patility 
Rt 11 Film Carrier 
Cartridge/Cassette 
Aperture Card m 
Microfiche 
Frame Retrieval 
Shaded Screen 

Still Pictures 

Motion Pictures 

Television/video 

3-D <ObJects) 

3-D (Holography) 

Audio (Recording) 

Programmed fnstructipn 

Text 
CAi 

Demonstration 

Oral Presentetlon 

Hand-held Computers 

Imagery Enhancing 
Techniques 
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Considerable research in the education and training literature shows the 
advantages of various media in work environments intended to meet learning 
or performance objectives. A modification (Table A) of Allen's (1967) illus- 
tration of the relationship of instructional media to learning objectives 
is considered the most useful classification matrix for such a comparison. 
Notice that no media type can be ranked "high" in meeting the objectives of 
learning factual information, performing skilled perceptual motor tasks, or 
developing desirable attitudes. 

A final factor important to JPA design itself is the amount of effort 
expended on improving the aid's intelligibility (Shriver & Hart, 1975) or 
comprehensibility (Post & Price, 1974). This includes not only the read- 
ability of the written material but also the usability of graphics. 

Integrated JPA/Training/Job Design 

The importance of integrating the JPA, training, and job design cannot 
be over* iphasized. The three have been considered trade-offs, depending 
on cost, 4 wrf ormance, and manpower limitations in meeting overall weapon 
system/mission objectives. In the broadest sense, this is true. The avail- 
ablity of a highly proceduralized JPA (i.e., an FPJPA) may eliminate the 
need for formal training and allow work centers to better design the work- 
load around a complement of experienced and relatively inexperienced per- 
sonnel, but the FPJPA would probably cost more than a deductive JPA. Less 
expensive deductive aids, on the other hand, require greater training and 
on-the-job experience and require work centers h to operate primarily with 
highly skilled personnel. 

However, this seen*s to be a gross oversimplification of the relationship 
among these three technologies. First, no one technology (i.e., JPA, train- 
ing, or job design) is a direct substitute for the other. All three must 
be integrated for any weapon system. The savings suggested in the litera- 
ture for military training (Shriver & Hart, 1975) as a result of the FPJPA 
cannot be realized simply by reducing training. The longer-term personnel 
career considerations and the FPJPA's inadequacy for complex troubleshooting 
tasks must also be taken into account (Klesch, 1977), 

Second, the three technologies may have common cost areas. Much of the 
cost in developing a JPA is in the front-end analysis that, to a large extent, 
is fundamental to task-oriented training and job design. Job or task analyses 
designed to cover the requirements of all three technologies during weapon 
system design should be considerably less complex and costly than if each 
area conducted independent front-end analyses. 

Further, it appears increasingly obvious that, after major manning 
decisions have been made, the selection of a JPA technology should be made 
prior to the selection of the other two technologies. Task-oriented train- 
ing and work center organization can be more readily designed to support 
or take advantage of an aid than if the aid is designed to complement a 
style of training or work center organization. Also, an aid, once selected, 
is more similar to hardware in that it does not readily adapt to individual 
differences, whereas training and job design should by their very nature 
strive to support and take advantage of individual differences. The problems 
of trade-offs among the three disciplines is really one of systematically 
considering each of the technology areas at tie appropriate stages of weapon 
system RDT&E. 
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Note . Adapted frota Alien (1967). 



Training Interface with JPA 



The earlier studies, done for the Army by HumRRO (Shriver, 1960; 
Rogers & Thorne, 1965; Gebhard, 1970) provide the best data for Integrated 
job aiding and training between the two extremes of FPJPA with minimal or 
no training and conventional technical manual aiding with extensive formal 
training. On the M-33 antiaircraft fire control system, for example, the 
Army found that electronics technicians using FORECAST aids could perform 
as well as conventionally trained technicians even though their training 
was reduced from 30 weeks to 12 weeks (Shriver, I960). Similarly, Army radio 
relay/and carrier repair technicians using JOBTRAIN aids with a 50 percent 
reduction in training time were able to perform as well as conventionally 
trained technicians using conventional documentation (Gebhard, 1970)* 

A common question regarding the relationship of training and aiding 
asks what information is best presented via each technology. The literature 
provides numerous guidelines for assigning performance task coverage (Chalupsky 
& Kopf, 1967; Chenzoff, 1973; Lineberry, 1977). Information about the fol- 
lowing types of tasks would best be presented via a training technology; 

1. Tasks that are not very easy to learn on the job. 

2. Tasks that are hard to communicate with words. 

3. Tasks that need a great deal of practice for acceptable per* 
formance to be established. 

4. Tasks that allow little room for error. 

5. Tasks with serious consequences for errors. 

6. Tasks with reasonable training costs. 

7. Tasks that are performed frequently on the job. 

8. Tasks for which the required speed or response rate does 
not permit referral to a manual. 

9. Tasks performed by a large proportion of the individuals 
in a given specialty. 

10. Tasks that can be performed with considerable freedom in 
individual style. 

11. Tasks that are so new or undefined that the detailed steps of 
mastery are unknown. 

12. Tasks wherein psychosocial factors are important* 

An aiding technology would be the best method of presenting informa- 
tion about the following types of tasks; 

1. Tasks that are usually performed in a fixed sequence* 

2. Tasks that involve long or complex behavior sequences. 
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3. Tasks that are performed infrequently. 

4. Tasks that involve readings and tolerances. 

5. Tasks that can be rehearsed mentally before the need to 
perform them arises. 

6. Tasks that are aided by illustrations. 

7. Tasks' that utilize reference information such as tables, 
graphs, flow charts, and schematics. 

8. Tasks with branching step structures. 

9. Tasks remaining stable over long periods of time* 

10. Tasks for which established task sequences are critical* 

11. Tasks for which the performance method is likely to change. 
I2 t Tasks that must be treated with a very limited training 

budget* 

As discussed in the previous section, a distinction is sometimes 
made between tasks that need to be memorized (head) and those that can be 
aided (book). Maintaining a separation between maintenance training and 
aiding tasks, when the aid is an FPJPA and the training is "hands-on experi- 
ence," becomes extremely difficult. However, the difference between what 
is important to know without an aid and what is important to be able to 
do with an aid is still a good conceptual distinction. 

Job Design Interface with JPA 

Perhaps even more important than JPAs* effects on training are their 
effects on job design. Attempts to design a job or work center organiza- 
tion to take advantage of performance aiding have been few, although the 
potential manpower benefits are great. 

Post and Brooks (1970) developed a mathematical model of the F-AJ 
work center and applied it to the data collected under HAVAIR's AMSAS experi- 
mental study program. The findings demonstrated job guides' ability to aid 
performance on nontroubleshooting tasks for electrical, mechanical, and 
electromechanical equipments. In that study, 71 percent of the iaexperienced 
technicians* time was spent observing and assisting experienced technicians, 
and the other 29 percent could not be assigned. If job guides were used, 
then 83 percent of inexperienced labor's time could be spent on maintenance. 
The provision of job aids, therefore, could result in major job design alter- 
natives and flexibility for a work center. Post and Brooks* findings that 
a work center could expect a 52 percent increase in the availability of ex- 
perienced technicians, a 25 percent decrease in maintenance queues, and a 75 
percent decrease in failed maintenance actions, show that it would actually 
be feasible to increase operational readiness while reducing the number of 
maintenance personnel. 
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Horn (Note 2) demonstrated in field studies with FPJPA and Work 
Packages that a ratio of one experienced to four inexperienced technicians 
can perform all the maintenance actions of a work center. 

JFA/Training/Job Design Interfaces 

Several Important factors must be considered in job design inter- 
faces with JPA and training. The following factors have received the 
most attention; task organization, position definition, task modules or 
clusters, skill transfer areas, graded proficiency levels, span of super** 
vison, teaming, spares philosophy, equipment test philosophy, and the use 
of ATE. 

Skill transfer, for example, is always an important consideration \ 
in military operations. The ability to use personnel in operator or main- 
tenance tasks for which they have not been formally trained is especially 
important in combat or other emergencies. Enlarging the scope of a job for 
career advancement or to balance workloads on a day-to-day basis is also 
mora feasible if personnel can move readily from one job to another* Job 
aids 1 potential for facilitating such transfers has been demonstrated on 
several occasions. Rigney, Fromer, Langs ton, and Adams (1965a, 1965b) 
found that, with a fault locator aid (XFL), radio operators with little 
or no experience in electronics could readily solve radio problems* Horn 
(1972) also found that electronics and mechanical ratings could satisfac- 
torily perform each other's jobs on a mobile electric power plant with 
the help of FPJPAs. 

There has been some work to improve the methodology and provide 
guidance for determining an optimum JPA/ training/job design interface* 
In the model proposed by Post, Price, and Diffley (1976) and post (1977a) 
for selecting formats and media to present maintenance information, work 
center considerations such as span of supervision, personnel turnover, and 
maintenance demands (workload over time) were considered key decision criteria* 
It is unlikely, however, that data on those variables will be available early 
enough to make such sensitive selections on new system procurements* Here 
again, it appears that the decision for a particular type of job ali may 
be predetermined because of the personnel and training cost savings. 

Guidelines for position (or job design) and documentation interface 
have also been developed for the Bell Laboratories (Price, Post, & Barker, 
1973; Post, 1977b). These should be useful in the development of military 
systems to integrate JPAs, training, and job design for operation and main- 
tenance tasks. The following is a list of job design factors that have been 
shown to be important to work productivity (Post, 1977b): 



ERLC 



Factors 



Design Guidelines 



Proces _ s 
1. Sequence 



2. Dependencies 



3. Functional Integrity 



4. Workload 



5. Difficulty 



Personnel 
6. Complete Work 



7. Autonomy 



8. Comprehension 



9. Challenge 



10. Valued Abilities 



11. Meaningfulness 



12. Participation 



13. Career Potential 



Sequential tasks are gcod candidates 
to include in a job. 

Dependent tasks, even though they 
might not be sequential, are good 
candidates to include in a job. 

Tasks vith a dommon objective are 
good candidates to include in a job. 

The timing and duration of job tasks 
should be considered so as not to over 4 
or underload the incumbents. 

Task difficulty should be compatible 
with incumbents 1 performance capa- 
bilities and should be relatively 
homogeneous across the job* 



Job tasks should result in a sense 
of completion or closure* 

Job tasks should permit the incumbent 
to plan and control his own work. 

Job tasks should be understood (both 
how and why) to promote performance 
reliability* 

Job tasks should be challenging, yet 
within the incumbent's capability. 

Job tasks should require significant 
abilities (ski lis /knowledge) . 

The product of job tasks should be 
worthwhile and have a personalized 
nature (e.g*, "my equipment"). 

An incumbent should be able to relate 
his tasks to the organization's goal 
or product. 

Job tasks should provide direct or 
indirect opportunJ ties to learn new 
and valued skills. 
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JPA Test and Evaluation Technology 

Without a method for assessing a JPA's value relative to competing tech- 
nologies and to other JPAs, there Is no basis for knowing which JPA would 
serve best or whether or not a JPA should be used at all. Tools for making 
such an assessment generally can be classified as survey, experimental 
(laboratory), demonstration (field), and simulation. 

Simulation 

? JPA technology has been tested and evaluated extensively with all 

four techniques. Losee et al. (1962) and Jfoin (1974) are good examples of 
using the survey technique. Whether a survey employs a questionnaire, 
an Interview, or some combination of thfc two, It can obtain a large amount 
of data relatively quickly and Inexpensively. The survey can be used 
(1) to determine whether users would prefer a highly proceduralized format 
In pocket book form to a deductive format with all the Information on one 
page, (2) to guide the choice between a printed page or microform frame 
display at the user site, or (3) to save the cost of printing expensive ^ 
color-coded troubleshooting charts when other methods of coding would be 
just as effective. 

The preferred method of collecting data for use In decision making 
at all levels Is the controlled experiment, either In the laboratory or in 
the field. However, of all the JPA systems and techniques listed in the 
glossary, less than 25 percent have laboratory or field performance data 
to demonstrate their effectiveness Most of these* as Rowan (1973) pointed 
out, simply compared the techniques against "traditional" documentation* 
Only a handful of studies have compared more than two techniques or have 
attempted to Improve the conventional technique before testing. Studies 
that have attempted to improve the accuracy, completeness* co^>actness, 
and readability of the conventional technique include Post and Brooks (1970) 
on non troubleshooting tasks and Elliott and Joyce (1968) on troubleshooting 
tasks. Studies that have compared more than two techniques include Atchley 
and Lehr (1964) on proceduralized instructions versus data flow versus 
schematics; Horn (1973), on Augmented Action Tree (AAT) versus MDC versus 
conventional Ifavy technical manual; Potter and Thomas (1976), on FPJPA versus 
Logic Tree versus Air Force Technical Order; Booher (1975), on six types 
of procedural ized instructions; and Frey (1976), on three types of pictorial 
med la. 

Because of the difficulty and expense of collecting data experi-* 
mentally, computer simulation models are often considered as an alternative* 
A typical demonstration in the field may cost several hundred thousand 
dollars, the bulk of which goes for the preparation of JPA teat materials. 
Even a controlled laboratory experiment that deals with a few tasks or sub- 
tasks may easily cost $50,000 to $100,000. However, the data used to 
validate simulation models, even in fields other than JPA, are often so 
unreliable that simulation is seldom applied. 

The best applications of test and evaluation models have been system 
effectiveness estimates using simpler mathematical models. Post and Brooks* 
(1970) effort on manpower utilization is a good example. The only other 
study of this nature concerned the Air Force C-141A aircraft. The analyses 
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reported in the PIMO final report (Serendipity, 1969) and by Braid (1975) 
Indicate that simply reallocating time from OJT to maintenance would de- 
crease the total time a one-term enlistee spends In training by 25 per- 
centt Other predictions were that flight departure reliability could be 
Improved by 50 to 65 percent; operational readiness time, by 38 to 40 per- 
cent; unscheduled maintenance manpower, by 30 to 39 percent; and spares 
demands, by 30 percent* If these results — assuming that the conservative 
estimates made by Rowan (1973), Shrive and Hart (1975), and Shrlver (1977) 
are valid--could be fully applied to Navy personnel, training, and main- 
tenance support associated with electronic systems, then the Navy could 
expect (1) decreases In first-enlistment training costs of 50 to 80 per- 
cent, or $900 million annually, (2) savings in spares of $75 million per 
year due to reductions in erroneous removal, and (3) a decrease in main- 
tenance labor force of 15 percent, or $700 million annually* 

Other payoffs are projected, but they have no quantitative basis 
for evaluation; (1) an increase in the affective use of lower-skilled per- 
sonnel, (2) Improved job involvement and satisfaction among first-term en- 
listees, and (3) an increase in retention rates for productive personnel* 

One simulation technique for assessing and predicting the relative 
performance effectiveness of job performance aids was developed at North 
Carolina State University (Smillle, Edsall, Ayoub, & Muller, 1976), That 
approach combines feat ires of predetermined time and motion systems (PTMS); 
a simulation modeling algorithm, Graphical Evaluation and Review Techniques 
(GERT); and the Systems Analysis of Integrated Networks of Tasks (SAINT), 

Results of the simulation process should yield substantial Informa- 
tion on JPAs' effects on performance time and errors. For Instance, statistics 
can be obtained on the total performance time and the probability of success* 
fully completing the task while using a specific JPA. In addition, simula- 
tion results can be used to identify the task step or steps that might be 
crucial to the success or failure of the task at hand. Based on these 
statistics and the cost data associated with JPA production and maintenance, 
a cost/benefit analysis can be performed to determine the best JPA~malntenance 
system or design— a design that could be expected to yield minimum performance 
time and errors at minimum cost. 

Unfortunately, the model is not in a form that can be readily applied 
to specific JPAs, training, or job designs. It appears to be a rather com- 
plicated way of analyzing relatively simple task steps, and the model was 
never validated. Model development and validation were attempted on. such 
simple tasks that time and error differences obtained in both the experi- 
ment and simulation were insignificant. Attempts to further reduce task 
steps Into specific behavioral processes would not be meaningful on such 
easy operations. Furthermore, no attempt was made to address troubleshooting 
tasks* where it now appears that the greatest difficulty for JPA selection 
lies. At present, therefore, no usable mathematical model or simulation 
technique exists for assessing or predicting the effectiveness of job per- 
formance aids. 
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CONCEPTUAL ORGANIZATION OF JPA TECHNOLOGY 



Development/Selection Methodology 

The previous two sections of this report described the research and tech* 
niques that comprise the current JPA technology base. Unfortunately,, the way 
JPA techniques have evolved and research has been conducted has not made the 
most efficient use of available resources. Also, most of the work was done 
without thought to fitting it into a larger conceptual framework that would 
later be useful to researchers, developers, and implementers. 

The development or selection of JPA techniques has usually been done 
by one of five methods: 

1. Trial and error . With this technique, which is the most .common, 

an existing aid form (most often a technical manual for equipment operation 
or maintenance) is gradually changed over time to meet user demands. Re- 
search data are seldom collected. General user acceptance is considered 
the primary measure of success. Although this approach is generally quite 
inefficient* it has yielded some very good JPAs (e.g., the NAVAIR Work 
Package concept, the General Motors Diagnosis and Repair Manual). 

2. Fi^t . Although commonly applied in the military services to procure 
JPAs, this is the worst method. Under it, a system is built and implemented 
by decree. This method is largely to blame for the problems of the NAVAIR 
and NAVSEA microform systems. 

3. Brochuremanship . Each of the 101 performance aids listed in the 
glossary has probably, at one time or another, been sold to a project manager 
simply on the basis of advertised advantages over the "conventional technique. w 
This is known as brochuremanship. 

4. Literature extrapolation . A few performance aids (e*g*, the FPJPA 
type shown in Figure 1) have been developed by extrapolation from the litera- 
ture. These incorporate such features as short sentences, short words, verb 
lists from readability research * and the limitation of seven callouts per 
page. This method provides a good quality system but may result in higher 
costs than necessary. 

5. Algorithm . The technique stimulating the greatest interest recently 
has been that of building and selecting the optimum aid form, considering 
(a) the combined effects of several variables on user performance and (b) 
the cost factors. There have been a number of attempts to develop models 

or algorithms for this purpose, but no aid forms have actually been con- 
structed using these techniques. Inherent in this approach is the assump- 
tion that selecting the optimum JPA depends on simultaneous consideration of 
major variables such as environment, personnel characteristics, and type of 
task, much as diagramed in Figure 6, a conceptual model for JPA technology 
utilization (Blanchard, 1977). 




Equipment/ 

Environment 

Characteristics 



1 






j 


1 Id V 






L 


X <v 



Personnel Characteristics 



bo 



Specific 
JPA 

Approach Mode 



Figure 6, JPA selection variables. 
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None of the techniques, including the algorithm, has provided a satis- 
factory way of organizing JPA technology for research or implementation 
decisionst For that reason, an attempt was been made to formalize an 
organization of JPA technology that might be conceptually useful to those 
employed in either applying or advancing the current technology t Hope- 
fully, the concept presented here will help organize the available tech- 
nology for easier application decisions and for better determination of 
which future research should be emphasized* 

Organizing the Model 

A simple model for organizing JPA technology was found useful for re- 
viewing the techniques listed in the glossary, for appraising the various 
algorithms recently developed in other military programs* and for identify- 
ing the known behavioral concepts applicable to JPA basic research* Briefly* 
as illustrated in Figure 7, this model consists of three major JPA tech- 
nology levels* each of which has a number of distinguishing characteristics* 
The levels are (l) the JPA system level, (2) the performance aiding component 
level, and (3) the performance aiding element level* Depending on which 
level is addressed, there can be considerable variation in the performance 
aid structural form, research questions, application questions, and types of 
expertise involved. 

Two important considerations become immediately obvious from this organiza- 
tion. First, Figure 6 f s three-dimensional arrangement of variables is use- 
ful only after one decides what level is being addressed* Applying variables 
directly to JPA systems, components, and eler.~nts alike without identifying 
what part of the larger system is being described simply adds to the exist- 
ing confusion. Second* the organizing model provides a visual reminder of 
the major differences between the optimum paths for (l) the design of JPAs 
and (2) for the application of JPAs in personnel systems. The design build* 
ing path should move from the element level to the system level, whereas 
the critical factors in JPA application diminish in importance as one moves 
from the system level to the element level. The structure of the three 
different levels is detailed in Table 5, 
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JPA SYSTEMS 
JPA COMPONENT 
JPA ELEMENT 

JPA TECHNOLOGY LEVELS 

Figure 7. JPA technology levels. 
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JPA Organizing Concept 
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Questions 
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Format interface? 
Content coverage? 
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Ratio of graphics to 
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Research Questions 


JPA/training/job design? 
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Operational effectiveness? 
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Single/multiple-channel 
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Readability? 
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JPA Systems Level 

A substantial body of literature describes two-way comparisons 
between new aiding concepts and conventional practice. Summaries (Rowan, 
1973; Shriver £ Hart, 1975) almost always show that aiding enhances on-the- 
job performance. Some job aids simply allow skilled people to do better, 
some allow unskilled people to do jobs they couldn't otherwise do,' and some 
allow jobs to be done better or the seme with fewer people. The selection 
of one of these types depends on factors that must be treated at fairly 
gross levels (e.g., Troubleshooting tasks versus nohtroubleshooting; low, 
intermediate, or high technician aptitude; high versus low equipment com- 
plexity). Data at this level are primarily useful for JPA systems decisions. 
An attempt is generally made to provide JPAs that cover all or large portions 
of several task groups required to support a particular hardware system or 
equipment. It therefore appears that the term "job" is most appropriate at 
this level. Implementation usually requires one specification for development, 
test, and evaluation, and another for associated training requirements * 
Good examples of JPA systems are the Ait Force's Fully Proceduralized Job 
Performance Aid (FPJPA), ttie Army f s Improved Technical Data and Training 
(ITDT), and the Navy f s Functionally Oriented Maintenance Manual (FOMM) . 

JPA Component Level 

The JPA system is generally comprised of several different types of 
presentation styJp* or formats, each designed to aid a different category 
of behavioral t,isks (e.g., lubricate, remove, fault isolate, test). The 
FOMM, for example, contains tables, lists, functional blocks, matrices, 
written procedures, pictures, and schematics, each designed to aid different 
aspects of the maintenance tasks of any particular equipment. These perfor- 
mance aids, when used to present information, are in forms that tend to 
occupy the user's total field of view at any one time. Commonly described 
as Pages, frames, or display events, such units exemplify the component level 
of JPA technology, 

JPA Element Level 

In many instances, the presentation component is used to aid a large 
number of different behavioral sub tasks. Consider, for example, components 
such as an illustrated parts breakdown diagram or an electronic schematic. 
Specific behavioral subtasks (e.g., using callouts, searching for functional 
areas, following electron flow, reading voltages, reading wave forms) may 
be aided by different features of the same presentation component being used. 
Many times, features tfnt show positive performance advantages by themselves 
interact negatively with positive performance features of other aiding 
techniques at this level. A good example of this is the attempt to put 
on one page a large number of callouts to a figure* There can be good per- 
formance advantages in eliminating references to other pages, buL performance 
disadvantages will soon arise due to perceptual difficulties in isolating 
the correct callout. This level, the foundation for the component and system 
levels, is concerned with organizing information within a i>age, slide, film 
strip, etc. Investigation at this level most closely resembles conventional 
psychological experimental research in a rontrolled environment. This is 
also the level where the greatest amount of extrapolation from the experimental 
literature has been attempted. In new JPA development, caution should be 
exercised because laboratory data have seldom been collected on tasks that 
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resemble the actual operator-maintainer task. (For example, results from 
research on perceptual recognition, reading speed, and reading comprehension 
are almost useless for application to the understandablllty of short, pro* 
ceduralized Instructions.) Furthermore, almost no data deal with the 
numerous" Interactions among the behavioral subtasks involved in using any 
performance aid presentation Element. 

Decision Making at the JPA Levels 

Table S presents some examples to illustrate decision making at 
the three JPA technology levels. The different objectives and anticipated 
results depend greatly on which level is being addressed. In a long-term 
research program, the general direction in the development of a technique 
is from right to left in the table. In an implementation program, however, 
the crifcicality of the decisions in terms of cost/performance trade-offs 
and critical time for decision making is from left to right. 

Application questions at the system level might be, "Are a task 
analysis or special training requirements necessary?" or "What manpower limi- 
tations should be considered?" At the component level, the questions revolve 
around formatting interface, content coverage, and requirements for special 
information storage and retrieval equipment. Research objectives vary from 
identifying trade-offs among major technology areas at the system level to 
determining principles of technical material readability at the presentation 
element level. 

Perhaps one of the most useful applications of this concept is in 
identifying the decision makers who play major roles at the different levels. 
Top management decides when and where to use the overall JPA system, while 
personnel and training organizations decide which career paths can be made 
available with any particular aiding concept. However, top management should 
allow flexibility for the managers of specific programs to select the best 
combinations of schematics, illustrations, and special media to meet unique 
mission and environmental requirements. On the other hand, decisions already 
made at the system level and principles developed at the element level will 
limit the choices to be made at the component level. For example, if the 
system selection is FPJPA for personnel and training reasons, then the com- 
ponent choices are pretty well limited to illustrations, logic trees, and 
written procedures. Therefore, the program manager decision-making level 
that most JPA selection algorithms have been attempting to address may not 
have as critical a need for selection criteria as the other two levels. 

JPA Organizing Concept Applications 

There are at least three important application areas where the JPA 
organizing concept appears useful for grouping the theoretical and experi- 
mental literature: (l) cost trade-off analyses among performance aids, (2) 
JPA selection algorithms, and (3) theoretical trends in performance aiding. 

Cost Trade-off Analyses 

One of the first applications that can be made with the JPA organizing 
concept la in grouping the theoretical and experimental literature for coat 
trade-off analysis* At the systems levels, decision makers should look for 
data that support major trade-offs among technologies as well aa among 
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aiding systems. The data on the FPJPA system strongly indicate that little 
or no preassignraent training is required if the highly proceduralized aid 
form can be used. Less proceduralized systems, such as the Army's FORECAST 
(Shriver 4 Trexler, 1965) and JOBTRAIN (Gebhard, 1970), show a 50 percent 
reduction in training with joint aiding and training development. One field 
researcher (Horn, Note 2) obtained data that suggest ihrl major job design 
alternatives are possible with proceduralized aids. Horn's results showed 
that if the organization permits, one skilled technician can keep four 
inexperienced technicians busy through the use of job aids. One. of the 
few comparisons that - included more than one JFA system alternative was 
recently completed by Potter and Thomas (1976). Table 6 shows that LTTA, 
which is not as fully proceduralized as the FFTA but is considerably cheaper, 
shows performance advantages over conventional techniques very similar 
to the advantages of the FPTA. A cost/performance trade-off analysis 
from these data could easily lead to an organizational decision in favor 
of the LTTA over the FPTA as the optimum aiding system. 



Table 6 

Comparison of Three JFA Troubleshooting Systems 
for Performance Accuracy, Parts Consumption, and Speed 



System 


Accuracy 3 


Parts Consumption^ 


Speed C 


FPTA 


95.4 


.72 


30.8 


uTTA 


89.4 


1.05 


34.8 


TO 


60.6 


2.67 


59.9 



Note. From Potter and Thomas (1976). 



^ean proportion of problems solved. 
^Mean number of parts used incorrectly. 
c Mean time in minutes. 



There are few good studies that give data on component-level compari- 
sons. One notable exception is Atchley and Lehr (1964). Their data, shown 
previously in Figures 4 and 5, fit well at the component level because compari- 
sons are made of several distinct presentation components and varying levels 
of complexity. These show clearly the advantages of proceduralized instruc- 
tion over data flow diagrams and standard schematics. Unf ortunately t this 
technique is also the most expensive of the three. 
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Table 7 summarizes a comparison of picture/word combination formats 
(Booher, 1975) that might fit best at the eicnscnt level. Several subtasks 
were measured for performance time and errors in looking for the optimum 
organization of pictures and words within a component (in this case, a pro* 
ceduralized instruction). The data clearly indicate that Format IV, the 
pictorial/related word format (see Figure 8 for an example), is optimum for 
both performance measures; however, it may not be the most cost-effective 
(Booher, 1975). Formats III and IV (the pictorial formats) are about equal 
in price, at least double the cost of Formats I and II (the verbal formats). 
Format III, the completely pictorial format, can be eliminated on performance 
factors, leaving I, II, and IV. Surprisingly, Format I (with no pictures) 
may be adequate if time is not a major consideration, whereas Format II 
(Figure 9, from Booher, 1972), the one most like the best FPJPA and PIMO 
formats, would not come out particularly well in a trade-off analysis be- 
cause there are few instanced in which it is desirable to sacrifice accuracy 
for speed. 



Table 7 

Speed and Accuracy of Navy Maintenance Personnel Completing a Series of 
Behavioral Subtasks Using Procedural ized Instructions 
with Varying Picture/Word Combinations 





Speed 


Accuracy 


Format 


(in minutes) 


(Z Errors) 


I. Printed Words 


45.9 


25 


II, Printed Words/Related Pictorial 


38.8 


32 


III. Pictorial 


31.2 


34 


IV. Pictorial/Related Words 


30.7 


24 



Note. From Booher (1975). 
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Figure 9. Highly printed words, related pictorial format for 
fully proceduralized instructions. 
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JPA Selection Algorithms 

Another application for the JPA organizing concept Is In organizing 
the various algorithms or models for selecting and designing optimum perfor- 
mance aids. In doing so, It appears that the model of Post (1977a) on formats 
and media and tha^ of Shrlver and Hart (1975) on fundamental elements are 
system-level concepts, while the user^data match model developed recently 
by the Navy Technical Information Presentation Program (NTIPP) (Hughes, 
1977b) Is directed specif legally to the component level. A selection and * 
organizing model being developed by Bell Laboratories (Dever, Note 5) may 
fit best at the element level. 

r 

In selecting and designing optimum performance aids there are 
several problems that seem; unique to each level and that should be emphasized 
In their respective algorithms. At the system level, for example, the funda- 
mental elements that make jup front-end analysis are extremely Important. 
Shrlver and Hart (1975) pointed out that In designing an FPJPA It Is not 
enough to do the engineering analysis or even the human factors task analysis. 
If troubleshooting Is Involved, then a functional analysis Is required as 
well; and If the JPA Is to be used by Inexperienced personnel, then a be- 
havioral task analysis Is also required. Table 8 Illustrates the different 
levels of front-end analysis that aYe Involved, depending on the particular 
format and content selected at the system level. But even these analyses 
are Inadequate for an Integrated JPAVtraining/job design system. In the 
JPA conference at NAV PERSRANDC EN (Booher, 1977a), It was noted that trade- 
off analyses among the various disciplines supporting weapon system operation 
and maintenance must also be conducted as early as possible In the system 
development. These analyses should then be reiterated throughout the system 
life cycle. The earlier and the more completely that cost/performance analyses 
are done, the more likely It Is that maximum benefits will accrue from each 
of the three major disciplines. 

General rules of information presentation are best applied at the 
JPA system level. Post and Price (1974), In applying their organization 
principles to maintenance task technical Information manuals, recommended 
that designers should also attempt (l) to develop a maintenance manual 
on a hierarchy of maintainable units, (2) to Include a repair cycle over~ 
view that gives a diagrammatic representation of the succession of mainten- 
ance actions, (3) to consolidate all Information concerning a single main- 
tainable unit Into a single package, and (4) to standardize the content, 
format, and sequence of Information within a package. 

An algorithm for selecting optimum job aids at the component level 
Is considerably different. Often, the designer will have an opportunity 
at the presentation component level to choose from several competing formats 
for conveying the Information relevant to certain tasks. The choice could 
be within the print medium (I.e., table, chart, prose, or pictorial), or 
it might be among audio or audiovisual media. Decisions at this level vary 
with weapoit systems and tend to be best made by the program manager, pre* 
ferably with the assistance of presentation principles guidelines. Aids 
at this level tend to be structured Into composite wholes capable of being 
presented within the user's field of view. Several attempts have been made 
to classify the various presentation components (Curran, Note 6; Powers, 
Dickerson, & Nugent. 1977; Hughes, 1977b). 

5fc>'* 



Table 8 

JPa Format/Content Cost Factors 



Format & Content 


Chui piaenc 
Complexity 


&<iuiput£n t 
Type 


Equipment 
Analysis 


rune c lonai 
Analysis 


lastt 
Analysis 


Behavioral 
Jiastv 
Analysis 


■in ceii lgiQinty/ 
Comprehensibility 


rt lilting/ 
Distribution 


FPJPA (Troubleshooting) 


X 


X 


X 


X 


X 


X 


X 


X 


Partially Proceduralited 
(TS) 


X 


X 


X 


X 


X 




X 


X 


Simple Logic Deductive 
(TS) 


X 


X 


X 


X 






X 


X 


System Description (TS) 


X 


X 


X 








X 


X 


r rjrA vnoncrouoiesnooc ingj 


Y 
A 


v 

A 


v 
A 




v 
A 


Y 
A 


Y 
A 


v 
A 


Partially Procedural i zed 
(N-TS) 


X 


x 


x 




X 




X 




Systems Description (N-TS) 


X 


X 


X 








X 


X 


FPJPA (Complex Action) 


X 


X 


X 


X(TS) 


X 


X 


X 


X 


Standardized Operating 
Procedures 


X 


X 


X 




X 


X 


X 


X 


Periodic Action Aids 


X 


X 


X 




X 


X 


' X 


X 


Time Critical/ 
Hazardous Aids 


X 


X 


X 




X 


X 


X 


X 


One Trial Learning Aids 


X 


X 


X 




X 


X 


X 


X 


Highly Complex/ 
Integrated Systems 


X 


X 


X 


X 


X 




X 


X 
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The question of which JPA (e.g., table, chart, schematic) is optimum 
for presenting fault isolation information can also be asked best at the 
component level* However, existing data are generally scarce and inadequate 
for malting simple comparisons of JPA presentation components. Any decision 
about what is the best presentation component for any particular task and 
for any particular personnel attribute must be extrapolated from other data 
or "expert judgments." Hughes (1977b), for example, in developing extremely 
elaborate matrices of task actions versus "p resentat l on components 11 for 
several Navy ratings, was unable to cite a single "ideal" presentation 
component for any specific task baaed upon data from controlled experiments* 
Specifications and guidelines for component selection for a' particular 
task are also generally unavailable at this level* 

As of this ^writing, the Bell Laboratories study (Dever, Note 5) 
was not available for review. According to Dever's description, however, 
Bell's structured programming technique appears to be useful as an algorithm 
at the element level. It has been observed that many of the "comprehensibility" 
problems facing the designer of computer programs are similar to those of job 
aid designers. Consequently, much of ^what is already known about writing com- 
puter programs can be applied to proceduralized job aids. 

Theoretical Trends 

The JPA organizing concept may also help to group the theoretical 
trends most applicable to performance aiding. It seems that the literature 
on laotivation theory as applied to work and work situations™theories of job 
enrichment, job enlargement, and vertical and horizontal loading—are most 
applicable at the system level. Collins (1977) suggested that the experiential 
learning model and the action research method, with its foundation in Lewinian 
theory, can be suggestive of new developments in performance aid technology. 
These newer concepts, as well as older ones such as those of Miller, Galanter, 
and Pribram (i960), show how people develop hierarchical plans and strategies 
in organizing and executing their activities, and can best be applied to the 
integration of JPA systems into the larger organization* For example, the 
theory of Miller et al. suggests that people, while Welcoming tlje organiza- 
tion of information they must filter to plan their own activities, object 
to being tightly regimented in planning, organizing, and executing their own 
behaviors. In proceduralized troubleshooting this objection to rigidity 
might be expected to increase from the novice technician who welcomes the 
procedure (because there is no other way he could do the job) to the experi- 
enced trouhleshooter who might take serious exception to a particular pro- 
cedure (because of the increased demands or because it does not allow for 
his greater knowledge of possible system faults). This points out the 
necessity for future JPA systems to be sufficiently flexible to change in 
response to personnel career progression. 

At the component level, the theoretical and experimental base seem 
to be rooted most firmly in information communication theory* The work done 
in education and training on audiovisual channels of communication is applic- 
able, particularly when organized under Hartman's (1961) model for multiple- 
and single-channel communication* Hartman said that, since the advent of motion 
pictures and television, researchers in communication have tended to define 
channels according to the kind of information presented. In multiple-channel 
communication, the different channels may present redundant information, 
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related information, unrelated information, or contradictory information. 
In JPA design, multiple channels are desired only if the information in the 
two channels can be presented in redundant or related fashion, and with a 
high degree of confidence in low interference between channels. 

In single-channel presentations, performance on tasks aided by visual 
formats has been found to be superior or equivalent to tasks utilizing 
audio formats (Elliott, 1966; Severin, 1967; Golf et al., 1969). Written 
formats are more accurate than pictorial (Dwyer, 1968j Booher, 1975) but 
pictorial formats take less time to use (Dwyer, 1968; Dickey & Schneider, 
1971; Booher, 1975). The usefulness of pictorial information in facilitating 
perfo -mance in concrete operational tasks appears to be relatively indepen- 
dent of personnel attributes. In one experiment with Navy maintenance per- 
sonnel from 10 job skill categories, the best performance was achieved when 
the pictorial channel was used as the primary channel of a multiple-channel ; 
format (Booher, 1975). Similarly, studies with college students, both hear- 
ing (Dwy< 1968, 1972) and hearing-impaired (Reynolds & Rosen* 1973), have 
shown advantages in the comprehension of individuslized narrative presenta- 
tions with the use of highly pictorial formats. 

Lewis and Cook (1969), in their theory of telling (which em hasizes 
one-way communication between the user and the performance aid), could be 
useful to investigators at both the component and element levels. In their 
words, W A theory of telling might be equivalently described as a theory 
of clear communication. Ideally, it would specify the conditions under which 
information could be communicated, in one direction only, without ambiguity. 
It would spotlight the difficulties that frustrate the unambiguous transfer 
of information, and it would serve as a reminder that problems of communication 
are problems of unclear communication. w 

Chapanis (1965), in addressing the problem of following simple instruc- 
tions, pointed out how few data are available for decisions that might be 
made in designing instructions. At that time he was able to find only one 
experimental study that addressed the problem'of the understandability of 
the information being presented. A clearly formulated theory of tell: T ng 
that concentrates on understandability might be most beneficial for guiding 
further JPA research. 

One of the most perplexing research problems at the presentation com** 
ponent and element levels is that of controlling content. It is very dif- 
ficult to ensure that, content does not change when formats are varied. In 
fact, much of the research in multiple-channel comparisons has been deficient 
in this area. Many examples can be found in the literature where one type 
of audiovisual display was compared with another, but the results were value- 
less because totally different information was presented in each display. 

At the JPA element level, experimenters and practitioners have come 
to realize that there are serious limitations in extrapolating from the 
large experimental data base on perception, memory, learning, retention, 
motivation, readability, and so forth to the composite wholes "(presentation 
components) that are dictated by realistic operation and maintenance tasks. 
Since the data were seldom collected on tasks resembling operator or main- 
tainer tasks, extrapolations require considerable faith that the job task 
being presented in s JPA component can' be adequately defined by a summation 
of behavioral subtasks. 



Several researchers (Newman, 1957; Folley, 1961) attempted to Identify 
potential JPA techniques throush a taxonomy of behavioral subtasks, but this 
does not appear to be a particularly frultfil approach. The listing of each 
sub task becomes an extremely complex way of defining the various behaviors 
Involved *n job performance. Also, the data linking the various behavioral 
subtasks to specific job aid formats are extremely limited, and the complex 
interactions of behavioral subtasks for any realistir *"isks are undefined. 
Studies that have succeeded in finding optimum forms for realistic tasks 
(Dwyer, 1968, 1969; Booher, 1975) have varied format across wide dimensions 
while maintaining fairly gross categories of behavioral tasks. Furthermore, 
even if these problems could be overcome, the final JPA format designed to 
optimize each behavioral subtask would probably be rejected either as 
being too voluminous or too densely compacted, depending on the degree of 
proceduralization require J from earlier decisions at the JPA system level. 

An optimum format size for guiding operator maintainer actions 
can usually be assumed to lie somewhere near the totally verbal print or 
the completely schematic. Any deviation from these two extremes /ill gen- 
erally require greater space and be relatively more difficult to prepare, 
thereby increasing the size of the format and cost. Decisions to carry 
optimization to the level of the behavioral subtasks must be approached 
cautiously, therefore, with these factors in rnind. 

Perhaps a totally new theoretical approach is needed at the element 
level. Concepts in memory would seem to be fundamental to an understanding 
of optimum ways to organize information at this level. Smillie (Note 1), in 
a recent doctoral dissertation on performance aiding, used memory concepts 
such as encoding specificity, cue-dependent forgetting, episodic memory, 
and depth of processing to explain past research findings; but concepts 
in memory (along with theories of perception and reading) have yet to be used 
satisfactorily in formulating and testing hypotheses for performance aid 
research. 
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NEW DIRECTIONS IN PERFORMANCE AIDING RESEARCH 



Use of Memory In JPAs 

The theoretical discussion of JPAs by Frase (1976) and Smillie (Note 1) 
may be beneficial for researchers and JPA designers at the presentation 
component or element levels. Frase described a model of the information 
processing skills involved in reading, with an emphasis on memory in reading 
performance. Smillie suggested that theoretical developments in memory be 
used for hypothesis formulation and testing of JPAs. Furthermore, Smillie 
concluded that the most important concepts contributing to a theoretical 
base for understanding JPA performance differences in print and pictorial 
channels are (1) pictorial superiority, (2) encoding specificity/cue-depen- 
dent forgetting, (3) episodic memory, and (4) depth of processing. 

There is considerable agreement among researchers that pictorial stimuli 
are processed faster and remembered better than their verbal equivalents. 
Numerous explanations have been made to account for this fact. In stimulus- 
response terms, Klatsky (1975) concluded that urccessing is faster for pictorial 
stimuli because the process of forming symbolic representation." from verbal 
inputs is bypassed. Other explanations for the relative superiority of the 
pictorial channel are (1) that the array search time is reduced (Pavio & Begg, 
1974; Mandler & Johnson, 1976), (2) pictorial material may be encoded both 
aurally and visually (Pellegrino, Siegel, & Dhawan, 1975, 1976), and (3) 
the sensory code for pictures seems to be more differentiating and less 
susceptible to interference from successive items (Nelson, Reed, & Walling, 
1976). This does not mean, however, that pictorial JPAs will necessarily 
be comprehended better than verbal JPAs. There are other aspects of the 
storage and retrieval process that have to be considered before any such 
generalisations can be drawn. 

Tulving (1974) and Tulving and Thomson (1973) discussed the perceptual 
encoding and forgetting aspects of storing infonsaUon for later retrieval. 
They noted that the retrieval process depends on retrieval cues that are 
most precise in accessing previously stored information. The availability 
of retrieval cues in memory depend upon (l) how well during encoding the 
performer perceives the cues necessary for later response, snd (2) how 
resistant the relevant cues are to forgetting. JPAs designed to take 
advantage of this memory process would strive to provide cues in the presen- 
tation format that (1) are most likely to be perceived as relevant for en- 
coding an aids for later information retrieval, and (2) are most likely to 
trigger retrieval cues at the time of performance. Smillie implied that 
specific verbal cues, rather than pictorial, would lead to greatest perfor- 
mance accuracy under this concept. 

Episodic memory and depth of processing concepts may also be important 
to JPA design if information inputs are truly stored as unique traces (e.g., 
phonemic, semantic, imaginal) and if .the memory traces are stored as specific 
episodes* Smillie interpreted this to mean that each instruction in a pro- 
ceduralized JPA should be designed to provide individuals with unique episodes; 
that is, to avoid miscuing and interference, each inotruction step should be 
as distinct as possible from the rest of the instruction. The depth of pro** 
cessing of each unique trace tends to be a function of the amount of time 
and the level of analysis 'undergone while attending presentation formats. 
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It is generally easier for a designer of procedural instructions to provide 
unique episodes and deeper encodings of information with verbal formats. 
These developments in memory, when looked at in a performance aiding context, 
may help to explain why both pictorial and verbal information must be used 
in varying combinations to maximize performance advantages. 

Towards a Theory of JPA Format Mix 

The theoretical discussions of memory and cognition and of multiple- and 
single-channel perceptual modes have been useful in interpreting unexpected 
findings in JPA research, but they have not been particularly useful in 
predicting optimum format design or in formulating JPA research hypotheses 
for experimental verification. For example, the cue summation and stimulus 
generation concepts used by Hart man (1961) and other audiovisual researchers 
to resolve discrepancies in findings regarding the learning efficiency of test 
materials does not explain the results of JPA picture-word studies. 

What seems to be most needed at all JPA conceptual levels — system, com- 
ponent, and element— is a theory for mixing JPA techniques, principles, 
and methodologies. For example, it is now thought that the optimum tech- 
nique at the system level may be to combine elements of both the FPJPA 
and the more conventional deductive aids into a hybrid aW (Post, 1977a; 
Klesch, 1977). It i3 also interesting to note that "pure" presentation 
formats (i*e,, prose, tables, pictures, and schematics) are seldom used 
alone to present information for realistic tasks. Printed prose or pro- 
ceduralized instructions usually refer to tables and illustrations for 
completing the information; even schematics will have some additional non- 
schematic information such as verbal cues or wave forms. In general, the 
knowledge that performance can be improved by combining print, pictorial, 
and audio channels is not nearly so valuable to JPA design as knowing what 
information to put in each channel and how to structure e^ch channel to be 
compatible with the other. 

Recent experiments on novel information transfer techniques for aiding 
location, comparison, and symptom recognition tasks (Booher, 1975), three- 
dimensional performance tasks (Frey, 1976), and long-term retention vSitterley, 
1974) suggest that major improvements in JPAs resulting fr^m changes in the 
structure of the format itself (as opposed to content change) are most likely 
to he found in designs that priniarily address the human visual memory system. 
However, the verbal channel must usually be kept available in a supporting 
role because of the ambiguity present in most techniques that attempt to 
convey performance information through visual memory. Alsn, the visual mode 
is generally considered inferior to the verbal in conveying abstract concepts, 
but even this does not appear true for all people. Split-brain research 
Indicates that a high level of abstract reasoning can be carried on in the 
visual memory centers (Diarn J & Beaumont, 1974), 

A start toward a theory of JPA format mixing may now be possible for 
the presentation component and element levels. Although a complete theory 
is beyond thi scope of this task, the experimental findings of JPA investi- 
gators are beginning to suggest a theory for JPA mix, Booher (1975) obtained 
the surprising results that the print channel alone was superior in perfor- 
mance accuracy to formats that most closely resemble the FPJPA, and that the 
FPJPA format was generally inferior in both performance time and accuracy to 
multiple-channel, highly pictorial formats. As Booher explained it: 
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The findings of relatively poor accuracy and time performance 
with high-print multiple-channel formats suggest that th* 
formats which most closely resemble conventional procedur- 
al ized instructions are not structured sufficiently well 
to take advantage of the distinct pictorial and verbal 
internal processes available for selecting, translating, 
and organizing information. (pp t 275-276) 

These findings now appear to be more universally applicable than was 
thought at that time. The same relative performance was also attained with 
hearing- impaired college students (Booher & Reynolds, in press). The find- 
ings here are even more surprising (i.e., the relative superiority of print 
format in performance accuracy over pictorial aid FPJPA formats) because 
hearing impairment usually seriously degrades reading ability, 

A different model than that proposed by Hart man (1961) or Smillie (Note 1) 
appears necessary to explain these unexpected results. It should first be 
recognized, as suggested by Haber (1970), that there are two different 
processing paths- pictorial and verbal— through the stages of selecting, 
translating, and organizing information presented visually. These paths 
can operate either independently or together, depending upon the informa- 
tion presented and the individual's approach to information comprehension. 
Applying this to JPA comprehension, it can be assumed that the pictorial 
path translates directly from pictorial inputs to the selection of a range 
of possible perceptual-motor routines for use in completing the task that 
organizes the routines in a time-sharing manner . The verbal path, however, 
translates from verbal inputs in a more step-by-step manner, tending to select 
routines in series. It is important to note that the human organism is capable 
of using both together in an optimum fashion but is forced to use one or the 
other initially depending on what type of format is presented. 

Before these experiments, the manner in which people are most predisposed 
to use the two systems for comprehending mixed formats was not known. How- 
ever, these results strongly suggest that people understand instructions best 
when the pictorial mode is used to aid in selecting and organizing a range 
of perceptual motor actions and when verbal material is available to confirm 
specific actions within the range. 

Another way of looking at efficiency in processing information through 
mixed formats is with the analogy of a two-stage filtering system: a first* 
stage, wide-band filter and a second-stage, narrow-band filter. The pictorial 
channel would be best for the wide-band activity, quickly reading the memory 
routines most likely to be applicable to the situation at hand, while the verbal 
channel, acting as the narrow-band filter, would be most useful in selecting 
the most accurate response routine available within the range obtained by the 
first filtering action. 

This concept may have applicability beyond the proceduralixed instruction, 
A recent study (Curran & Mecherikoff, 1978) of Navy maintenance personnel 
performing location and identification tasks using technical manual illuatra- 
tions suggested that the concept may also apply to deductive, nonprocedural ized 
aids. The study investigated the optimum arrangement and design of callouts 
consisting either of numbers, nomenclature, or nomenclature and numb era com- 
bined to aid in locating and identifying mechanical parts on cross- sectional 
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illustrations. In discussions before the data collection, an attempt wa s 
made to predict the optimum callout arrangement and mix of pictorial and 
verbal channels. It was reasoned that a wide-band filtering action would 
first be required to get the user to the correct area of the illustration. 
This could best be done with callout numbers and an arrow (a highly pictorial- 
channel technique). Often, however, it is not obvious, simply from viewing 
the illustration, exactly what the part is. A name (verbal channel) would 
be best to perform such identifications. In deciding then on an optimum 
format for callouts to aid in both locating and identifying ports, the theory 
suggests that the designer (1) use both callout numbers and arrows (pictorial) 
and names (verbal), and (2) arrange the numbers sequentially to be prominently 
displayed alone, with each name nearest its own callout number and arrow. 

Curran and Mecherikoff found that nomenclature and number callouts arranged 
as described above resulted in the best performance for identification tasks 
without degrading performance on location tasks. Some very ^useful future 
research efforts, suggested by these studies, could have both theoretical 
and practical merit. A particularly interesting research \area would be the 
optimum mix of verbal and pictorial information for schematics, functional 
block diagrams, and wiring diagrams used in maintenance troubleshooting 
tasks. Here again, the problem is not so much when to use a particular 
visual aid or even how to arrange the lines and blocks on a page. What the 
JPA designer needs to know is what kind of verbal information would best aid 
troubleshooting and where should it be placed in the chosen JPA formats. 

Priorities for Future JPA Research and Technology 

In R&D efforts at the system level, the greatest potential payoffs are 
seen in the efforts recommended at the JPA conference (Booher, 1977) and 
categorized as (1) JPA technology application, (2) methodological studies, 
and (3) future R&D, The current project plans to address the first two 
categories, including the R&D necessary for a long-term test and cost/benefit 
evaluation of performance aids in large personnel and training systems. Two 
of the papers presented at that conference (Collins, 1977; Shriver, 1977), 
provided specific recommendations for future JPA R&D at the system level 
that the current project does not address directly: (l) initiate research 
and development efforts to fill performance data gaps in the JPA technology 
base and to explore the feasibility of new concepts for JPA, and (2) develop 
a scientific and technical approach for identifying, developing, testing, 
and evaluating principles and methodology applicable to the implementation 
of advanced technology into Navy weapon and logiatics support systems. 

Research should be conducted at the systems level, therefore, with JPA 
aystem technology remaining relatively fixed while the effects on training, 
the personnel system, and job design variables are studied. In general, 
the most applicable methodology for implementing an integrated JPA/training/ 
job design paradigm should be that of an integrated personnel system, capable 
of self-design, test, evaluation, and change in response to advances in tech~ 
nology. Unfortunately, an integrated -personnel system implementation metho- 
dology does not currently exist, and efforts along these lines may need to 
look to a project such as the Job aids T&E for the data necessary for its 
own development. 
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More specifically, research on work, motivation, learning, and iesues 
of aiding versus other technological means for improving performance are 
especially important system considerations in successful JPA implementation. 
The methods and problems of integrating JPAs with training and job design 
should receive immediate attention, particularly with efforta in algorithm 
development. Also,' problems of career advancement; skills retention* and 
other major personnel system factors have not really been investigated 
for the effects of progressively changing JPA systems (e.g., FPJPA to hybrid 
to deductive). The optimum points in the personnel career paths for the 
introduction of higher level JPAs should probably be another major factor 
to consider in future algorithms and integrated personnel system trade-off 
models. 

In addition, there is still a need to determine the effects of aiding 
on lowered aptitude levels and to study the motivation to use aiding systems. 
Studies at the other two JPA conceptual levels may give some information about 
the ability of different aids to allow lower-aptitude personnel to perform 
better or to provide greater job satisfaction through on-the-job performance, 
but whether performance aids are to be used more than 5 percent of the time 
(as is generally reported for maintenance instructions) or whether lower- 
aptitude personnel can handle other aspects of the job that have not been 
aided are systems research problems. 

At the JPA component level, work on new aiding devices, basic comparison 
performance data? and further work on aids that change with personnel needs 
(e.g., hybrid aids) is still warranted. Host needed at this level are criteria 
for making valid comparisons that consider both cost and performance data* The 
biggest problem here is obtaining valid data on what it costs to develop a 
JPA. 

The research problems at the JPA element level are greatly in need of 
basic research. A whole new approach may be required to solidify the founda- 
tion of performance_^iding. The readability of technical instructions, for 
example, nee\^^€mething more than standards of short words and short sentences 
drawn frojn the reading research for education. Performance data on the com- 
prehensibility of technical instructions indicate that information transfer 
processing for decision making is on the order of 5 to 25 words per minute, 
whereas reading studies operate more on the order of 100 to 1000 words per 
minute. 

Finally, a problem that has yet to be addressed and that cuts across 
all three JPA levels is that of determining what kind of process is needed 
to produce and measure high-quality aids. If performance aids cannot be 
made extremely accurate at a reasonable cost, then the attempts to solve 
systems problems in motivation and productivity with lower skill and experi- 
ence levels are not going to be very satisfactory. Similarly, holograms 
with false images or picture-word formats with erroneous illustrations and 
verbal descriptions are of little value to the user. 

Perhaps what is. needed most, whether for improving performance aids, 
finding management techniques to raise quality, or compensating for in- 
adequate quality, are early warning systems with methods of predicting 
errors in the aid development process. 

< i 
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CONCLUSIONS 



Several JFA systems, classified as format/ content types, are sufficiently 
developed to be integrated with major military personnel and training systems 
with reasonable expectations of major payoffs in reduced personnel, training, 
and maintenance costs and in increased overall operational readiness. These 
basic types of JPAs vary primarily in accordance with the amount of pro- 
ceduralization required, which in turn is directly proportional to the amount 
of front-end analysis employed. The major cost items for comparing one format/ 
content JFA with another are the amount of detail required for front-end 
analysis and the amount of required graphic or verbal enrichment. The 
greater the amount of proceduralization and enrichment, the larger the size 
of the aid. 

Three format/ content types—the FPJPA, the hybrid aid, and the deductive 
aid—seem sufficient to cover the aiding needs of most operators and main- 
tainers throughout their Navy careers. With the exception of quality assur- 
ance, which is considered a delivery system problem, the PPJPA and the deduc- 
tive aid (HAVAIR Work Package or NAVSEA SIMM/FOMM) are fully developed 
systems. The hybrid aid is still in conceptual form, and probably needs 
further refinement before full-scale demonstration or implementation* 
This is particularly true for complex digital electronic systems tthere no 
satisfactory aiding techniques were found. 

Three other major types — display media, applied training, and peripheral 
diagnostic — are secondary means of improving performance through aiding. 
Systems that could be classified into these groups often had a format/ con- 
tent aspect that accounted for the increases in performance. 

The greatest weakness in JPA systems technology now lies in the systems 
for producing, distributing, and updating accurate JPAs. The only delivery 
systems presently available for providing format/content JPAs are improved 
technical publication systems. The Navy Technical Information Preaentation 
System (NTIPS) now under development should greatly strengthen this branch 
of the technology. 

Unique demands of the job environment (e.g., work space, wind, temperature) 
often create special physical design requirements for the job aid. In most 
instances, however, specially developed devices (e.g., audiovisual or micro- 
form) offer few advantages and several disadvantages to the user. Host infor- 
mation aiding needs obtainable within the state-of-the-art can usually be met 
with hard-copy print modified to fit environmental constraints. Devices that 
can inherently increase the capability of the user beyond that possible with 
hard-copy print (e.g., three-dimensional devices, interactive computer-aided 
devices) need further research to determine whether or not aiding technology 
can be advanced along these lines. 

Much of the recent JPA research by the Army, Navy, and Air Force has been 
directed toward the development of algorithms or models to be used in select- 
ing the optimum JPA. Several factors are important in the selection of the 
appropriate aid form. In most instances, the application of these factors 
to an algorithm for the selection of optimum performance aids is not com* 
plicated. The PPJPA, for all its promotion as a solution to the problem of 
decreased enlistment skills, does not appear satisfactory for aiding lower- 
aptitude personnel to do troubleshooting. However, inexperienced personnel 
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who meet the usual altitude levels can do troubleshooting with the aid of 
an FPJPA after minimi training. Also, the FPJPA will not satisfactorily 
cover all troubleshooting and maintenance actions, particularly those In 
complex electronic systems (both digital and analog). Such maintenance 
tasks (perhaps 25% of the maintenance workload) require additional training 
and less proceduralized aids. 

Once a set of aids for a particular weapon system or a particular rating 
has been selected, the preferred medium and other physical characteristics 
must be determined. Too often, such decisions are made first and other 
decisions must be made as compromises. Although there are exceptions, the 
general rule is to provide a hard-copy print medium wherever possible. 

Aiding technology stands on an equal footing with training, but the 
need to use lower-aptitude personnel and to cope with the high costs of 
training makes aiding a much more attractive technique for meeting most job 
objectives with first-term enlistees. Under current conditions in the military, 
it is more reasonable to save trainiag investments for those who are most 
likely to make the military a career and to emphasize aiding at the begin- 
ning of enlistments. Job design is different; it both leads and follows the 
selection of the JPA. In leading, it helps determine the actual need for 
aiding and how to plan which people in a work force should be aided, trained, 
or both. It follows by being applied in the work centers themselves. If 
particular JPAs are available, then the work center can plan its workload 
accord ingly. 

A conceptual organization of JPA technology has long been needed, as is 
evident from the piecemeal approaches to research and the turnover of experts 
in the field. Basic research is obviously needed to answer questions about 
the behavioral foundations of aiding. There is still a need to determine 
the effects of aiding on lowered aptitude levels and the motivation to use 
aiding. Cost/perf ormance criteria and basic cost data are needao to make 
valid comparisons of aids in trade-off models. Finally, research is needed 
to determine what kind of process is required to provide and measure high 
quality JPAs* 

After 20 years of development, performance aiding is now clearly a sought- 
after technology with clear directions for improvement. But it is not unusual 
for organizational inertia to limit promising advances before they can demon- 
strate their full potential. It remains to be seen if the military's personnel 
organizations can adjust to the impact that will accompany the introduction 
of such a sweeping technology as that offered by Job Performance Aids. 
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RECOMMENDATIONS 

1. The methods and problems of integrating JPAs with training and 
job design should receive Immediate attention in efforts directed toward 
algorithm development. 

2. Problems of career advancement, skills retention, and other major 
personnel system factors should be investigated to determine the effects of 
progressively changing JPA systems. 

3. Task-oriented training and job design should be developed to support 
or complement the aiding system. 

4. Aiding as a personnel system technology should act on an equal 
footing with selection and training, but it should be the primary technique 
for meeting most job objectives for new or first-term enlistees. 

5. Front-end analyses commonly conducted for integrated JPA and train- 
ing requirements should be expanded to include job design requirements. 

6. Selection of media other than hard-copy print should be based on 
system support requirements that improve the timeliness or accuracy of 
the aid rather than personnel performance-related criteria. 

7. Development of new aiding devices (e.g., hand-held calculators, 
holographic displays), aids that change with personnel needs (e.g., hybrid 
aids), and aids that specifically address digital electronic systems are 
needed to fill gaps and to advance the state-of-the-art in aiding tech- 
nology. 
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Acronym or 
Popular Name 



Full Title 
(If different) 



Source 



AAT 

ADMIRE/MIRAH) 

AOPEP 

AGCS 
AMSAS 

ATOMS 

ATS 
AVIS 

BFIC/BIFAC 

BLOCK FORM 

BRITISH ALGORITHM 
C-A LOGICAL PROCESSES 
CATA 
CMG 

Computer-aided Loop 

Diagram Representations 



Augmented Action Tree 

Automatic Diagnostic Maintenance 
Information 

Automated Data Preparation Evaluation 
Program 

Alden Graphic Communication System 

Advanced Manpower Concepts for Sea- 
Based Aviation Systems 

Automated Technical Order Maintenance 
Sequence 

Administrative Terminal System 

Audiovisual Information System 

Binary Fault Isolation Chart 



Computer-Aided Trouble Analysis 
Condensed Maintenance Guides 



BioTechnology, Inc. 

Westlnghouse Electric 
Corporation 

McDonnell-Douglas Aircraft 
Company 

Alden, Inc. 

Naval Air Systems Command 
Boeing Aircraft Company 



Letterkenny Army Depot 

Westlnghouse Electric 
Corporation 

Air Force Human Resources 
Laboratory 

British Air Force 

(Various sources) 

Vought Aeronautics 

Hughes Aircraft 

Technical Communications 



Acronym or 
Popular Name 



Full Title 
(if different) 



Source 



00 

o 



Computer Graphics and 
Visual Module System 

Calculator (hand-held) 



CONSD 

CONTEX 

COST 

DACOM 
DA TOM 

DSA 
DIMATE 

3-D Dynamic Display 
ELECT ROC ULAR 
FASTI 

FEFI/TAFI 

Ftlesearch System 
FIST 

FLAPS 



Personal Programmable Calculators 



Condensed Servicing Data 



Concentrated Odor Sensing Technique 



Data Communicator 

Data Aids to Training Operation 
and Maintenance 

Diagnostic Sonic Analyzer 

Depot Installed Maintenance 
Test Equipment 



Electronic Ocular 

Fast Access to System Technical 
Information 

Flight Engineers Fault I sol at ion/ Turn 
Around Fault Isolation 



Fault Isolation by Semi-Automatic 
Techniques 

Functional Layout and Presentation 

System 



William A. Fetter 

Texas Instruments or 
Hewlett Packard 

General Electric Company 

VITRO Laboratories 

Illinois Institute of 
Technology 

Letterkenny Army Depot 

General Electric Company 

Curtiss-Wright Corporation 
RCA 

(various sources) 

Hughes (Brown, H. T. ) 

National Security Industry 
Association 

McDonnell-Doaglas Aircraft 



U. S. Navy 



Hughes Aircraft Company 



Acronym or 
Popular Name 



Pull Title 
(if different) 



Source 



FOMM/SIMMS/IM/IMP/ 
IMC/BAMAGAT 



FORECAST 

FPJPA 

GH 

GPAM 

HAWK Radar Mechanic 

System Collection Manual 

Hayden Fault Indicator 

Holography 

Hybrid Maintenance Aid 
Imagery Enhancing Technique 
Implosion 

Information Retrieval and 
Display Systems 

ITDT 

JOBTRAIH 

JPA Assessment 
Algorithm 

JTG/JTM 



Functionally Oriented Maintenance Manuals 
Symbolic Integrated Maintenance Manuals/ 
Integrated Maintenance Package/ Integrated 
Maintenance Concept/ Block-a-Matic» 
a-Gram» a-Text 



Fully Proceduralized Job Performance 
Aid 

Diagnosis and repair manual 

Graphically Proceduralized Aids for 
Maintenance 



Integrated Technical Data and Training 



Job Task Guide/ Job Task Manual 



Naval Sea Systems Command 



Human Resources Research 
Organization 

Air Force Human Resources 
Research Laboratory 

General Motors Corporation 

Publication Engineers 

Human Resources Research 
Organization 

A* W* Hayden Company 

Randomline, Inc . 

Naval Air Systems Command 

Image tics* Inc. 

Hoehn and Lumsdaine 

(various sources) 

DAROOM 

Human Resources Research 
Organization 

Naval Air Systems Command/ 
North Carolina State 

DARCOM 



T 



Acronym or 
Popular Kane 



Full Title 
(if different) 



Source 



K INTEL 

LCI 

LGCP 

LTS-l/LTS-2 
M3DD 

MADAR/GPS 

MAGNACARD 

MAGNAVUE 

to MAINTSAlN 
to 

MDC 
MDS 
MEDIA 
KEMRI 

Kicrc-Vue 
KIN 

Mtnicard System 
MUHDATA 
Miracode System 
MTIPAC 



Learner-Centered Instruction 

Lexical-Graphical Composer-Pr inter 

Lincoln Training System 

Microfilmed Maintenance Manual 
Data Dissemination 

Malfunction Detection, Analysis and 
Recording System and Its Ground 
Processing System 



Maintenance and Training in Complex 
Systems 

Maintenance Dependency Chart 
Safeguard Maintenance Data System 



Maintenance Engineering Management and 
Repair information 



Kuiti- Image Microfilming 



Maintenance Management System 
Maintenance Requirements Card 



NSIA 
AFHRL 

Lincoln Lab MIT 
3*-M Company 

Lockheed -Georgia Company 
NSIA 

Magnavox 

Human Resources Research 
Organization 

Naval Sea Systems Command 

Western Electric 

Bell Labs 

Republic Aviation Company 

Republic Aviation Company 
Technical Operations, Inc* 

Martin Marietta Corporation 

Navy 

Navy 
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Acronym or 
Popular Name 



Full Title 
(if different) 



Source 



MT Magazine 
NAMES 

NEO STYLIZED MANUALS 
NTIPS/NTMS 

Optimum Picture/ 
Word Format 

PLMO 

Procedures Flow Chart 
PROFILE CARDS 
PS Magazine 

Pulsed Light Projection 

PYRA6RAM 

Quick Fix 

RAPIDS 

REPOM/MlRM 

RESTORE 
SADIE 

SHOCK ACTION 



Maintenance Tips 

Navy Aircraft Maintenance Evaluation 
System 



Navy Technical Information Presentation 

System 



Presentation of Information for 
Maintenance and Operations 



Pyramid Diagrams 



Rapid Automated Problem Identification 
Data Systems 

Reliability Prediction Oriented 
Maintenance/ Maintenance Instruc- 
tions Recorded Magnetically-^ 

Rapid Evaluation System to Repair 
Equipment 

Smiths Aural Diagnostic Inspection 
Equipment 



Navy 

Naval Air Systems Command 
Army 

Navy Ship Research and 
Development Center 

Naval Air Systems Command 
Serendipity* Inc* 



Philco Corporation 
Army 

Letterkenny Army Depot 
Hughes 

McDonnell-Douglas Aircraft 
Company 

Grumman Aircraft Company 
Naval Ordnance Laboratory 



Martin Marietta 

British European Airways 

Human Reaources Reaearch 
Organisation 
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Acronym or 
Popular Name 



Full Title 
(if different) 



Source 



4* 



SPARES Test Capability 
Report 

TASK! EACH 

TOPP 

Team Training 

TEAMS 

TRACE 

TROUBLE LOCATORS 
TRUMP /MI ARS 

TSP 
VAt 

Video file System 
VIDEOSONIC SYSTEM 
WALNUT 

WORK PACKAGE CONCEPT 

WSMAC 

XFL 



Task Oriented Plant Practices 



Transitor Radio Automatic Circuit 
Evaluator 



Technical Review and Update of Manuals 
and Public at ions /Maintenance Informa- 
tion Automated Retrieval System 



Versatile Avionics Systems Test 



Weapon System Maintenance Action Center 
Experimental Fault Locator 



General Dynamics 

J t W t Rigney/R* Fromer 
Bell Laboratories 
(various sources) 
Northrup 

Philco Corporation 

U, S t Air Forc<* 

Naval Air Systems Command 

Lincoln Lab MIT 

Naval Air Systems Command 

Hughes 

Letterkenny Army Depot 
Naval Air Systems Command 
McDonnell-Douglas Aircraft 
J, W t Rigney 



DISTRIBUTION LIST 



Chief of Naval Operations (0P-01X), (0P-01CR), (OP-401), (OP-964D), 
■ (OP-987), (OP-987H), (OP-99), (OP991B) 
Chief of Naval Personnel (Pers-lOc), (Pere*2B) 
Chief of Naval Research (Code 450) (4) 
Chief of Information (01*2252) 
Director of Navy Laboratories 
Commandant of the Marine Corps (Code MPI-20) 

Chief of Naval Education and Training (01), (N-2), (N-4) f (N-5), (OQA), (005) 

Chief of Naval Technical Training (Code 016), (Code N-3), (Cod* N-4) 

Chief of Naval Education and Training Support 

Chief of Naval Education and Training Support (001A) 

Commander Training Command, U* S* Pacific Fleet 

Commander Training Command, U* S* Atlantic Fleet (Code N3A) 

Commander, David W, Taylor Naval Ship Research and Development Center 

Commanding Officer, Fleet Combat Training Center,- Pacific 

Conanding Officer, Fleet Combat Training Center, Pacific (Code 00E) 

Comanding Officer, Fleet Training Center, San Diego 

Comnanding Officer, Fleet Anti-Submarine Warfare Training Center, Pacific 
Comanding Officer, Naval Education and Training Program Development Center (2) 
Commanding Officer, Naval Development and Training Center (code 0120) 
Commanding Officer, Naval Aerospace Medical Institute (Library Code 12) (2) 
Commanding Officer, Naval Training Equipment Center (Technical Library) 
Officer in Charge, Naval Education and Training Information Systems 

Activity, Memphis Detachment 
Officer in Charge, Navy Occupational Development and Analysis Center 
Officer in Charge, Naval Instructional Program Development Detachment, 

Great LaHes 

Director, Training Analysis and Evaluation Group (X&EG) 
Master Chief Petty Officer of the Force, U* S* Atlantic Fleet 
Master Chief Petty Officer of the Force, U* S* Pacific Fleet 
Master Chief Petty Officer of the Force, Naval Material COnoand (NHAT 00C) 
Master Chief Petty Officer of the Force, Naval Education and Training 
Command (Code 003) 

Personnel Research Division, Air Force Human Resources Laboratory (AFSC), 

Brooks Air Force Base 
Occupational and Manpower Research Division, Air Force Human Resources 

Laboratory (AFSC), Brooks Air Force Base 
Technical Library, Air Force Human Resources Laboratory (AFSC), Brooks 

Air Force Base 

CNET Liaison Office, Air Force Human Resources Laboratory, 

Williams Air Force Base 
Technical Training Division, Air Force Human Resources Laboratory, 

Lovry Air Force Base 
Advanced Systems Division, Air Force Human Resources Laboratory, 

Wright-Patterson Air Force Base 
Program Mueger, Life Sciences Directorate, Air Force Office of Scientific 

Rssearch (AFSC) 

Army Research Institute for the Behavioral and Social Sciences 
Military Assistant for Training and Personnel Technology, Office of the 
Under Secretery of Defense for Research and Engineering 

Director for Acquisition Planning, OASD(MRAiL) 
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Coooandant, National Defense University, Industrial College of the Armed 
Forces 

Director, Defense Activity for Non-Traditional Education Support 
Science and- Technology Division, Library of Congress 
Coast Guard Headquarters (G~P-l/62) 
Defense Documentation Center (12) 



ALU 
86 



